I didn't understand the core issue, is it that due to non-alignment of the on-disk data you have to do a copy at load time, but if you could guarantee alignment of the actual blob content then you could use it directly?
On a brief background read it looks like another issue is that blobs are not necessarily stored in contiguous regions of the file. Maybe there's some clever way to combine a VFS shim and virtual tables to store page aligned blobs in a region of the same file as the main database.
Don't pay too much attention to the inner-circle feedback. Some people subconsciously use the tone of reception to control you, especially peers/colleagues are prone to this. They see someone having a bright idea, and they genuinely don't like it just because it makes them feel small in comparison. The thing they do next - they try to extinguish the spark by creating an illusion of worthlessness in your mind. By doing so, they are getting rid of a potential competitor on their own way to success. According to psychology, they start feeling normal again by reducing you, this is a natural compensatory mechanism ingrained in human archetypes.
The key thing here is to be able to discern between real and manipulated input information. For this, gut feeling usually works best: it spots that whiny, attention-insisting, importance-seeking, fear-inducing tone of a manipulator, but oftentimes the higher-level nervous system of the brain suppresses that signal (e.g. "How can a well-educated and charismatic person feel a bit off and responsibility-lacking sometimes? It cannot be right, so it must be something with my interpretation of reality. I bet they have the best intentions.")
An even better approach may be not to search for feedback at all, unless it's shared naturally without any strings attached.
It’s a great insight and fun to think about. Perfect for HN.
If I put on my academic-paper-reviewer hat, I’d probably ask: what does that insight get us? What knowledge can we pull from databases into compilers? If we convert compiler/linker/loader operations into database lingo, then do we see patterns or more insights that let us resolve existing controversies/problems?
I haven’t read the author’s academic paper, so I don’t know what he considered the contribution to be. The contribution is “what knowledge was learned that improved the world”. The insight is enough for a minor paper, in my opinion. But if you can show it is the key to tackling an existing problem, that would be a great paper.
Academic reviews are rough. Hasty, inconsistent, often poorly written or incomplete. Responses can be random with good papers getting harsh reviews. Don’t take it too personally. Sometimes the information in a communication isn’t in the content - it is what it says about the communicator. Keep working on your writing and thinking clearly and your odds of better reviews improve … but will never be 100%!
I love your idea, 40% of which is because I had the same one. I think it is the next logical extension over using zip as the container (jar/ooxml/epub).
And it's already in use. In the geospatial world we have geopackage, basically a replacement for the still ubiquitous shapefile. It's build on the idea of the earlier spatialite extension for sqlite. Then, shortly after the introduction of geopackage, esri followed suit and released their 'mobile geodatabase', also based on sqlite.
It works great as storage for geometries and attributes and at the same time you have the full power of sqlite for all kinds of analysis.
Thanks for responding! I hope I didn’t come across as being overly critical. I’m curious what you would consider if designing a greenfield implementation of an object file format. What would you optimize for, and perhaps more interestingly, what would you not optimize for?
I like your phrasing; I think too often we get caught trying to optimize for everything.
For my own edification I often let performance & size slip to see if I can re-imagine something simpler from the start from a complete new direction;
This is often the greenfield hubris because it doesn't have all the same battle-scars of the full implementation so I try to make sure I take a very different approach to make it worthwhile :)
Then I just have faith that performance and size can always be improved over time.
That's an interesting idea as well.
Daniel wrote https://github.com/danielfullmer/nkv which also was a pretty fun experiment on how to improve data access within Nix. I love these cursed ideas.
Technically, you're right, it's a dynamic loader. Technically, it's pure dynamic loading.
If we look at the issue at its core, we're still a statically linked program in a hostile environment, forced to dynamically load device drivers from the system.
It's similar to Golang; on MacOS, it has to use libSystem, even though otherwise, these are the statically linked Go binaries we're used to and love.
Let me add a little more detail: if I use vdso with gettimeofday in a statically linked program on Linux, am I still a statically linked program, or not? :)
(author) -- fun to see this here.
I wrote this on the way (on the airplane) to DEFCON34 and Nix Vegas[1]
I didn't even get a chance to cruise around and see anything of DEFCON proper since the Nix community had me captive the whole time!
I gave 3 talks that I will update on my site[2] once they released.
