https://www.getdaft.io logo
Join Slack
Powered by
<https://gist.github.com/andrewgazelka/919bbd992e6...
# daft-dev
a
also I think your previous implementation did not read 4MiB into each slab as you just did one 
read
op
as far as I can tell
d
Even if you pass in a 4mb buffer to the read?
a
pretty sure... although not 100% positive
there are other methods like 
read_exact
macOS impl
image.png
yea looks like i am right
just one libc read call
d
I'll print out 
bytes_read
when I get home and confirm
A single read syscall can read up to 2gb though
a
@Desmond Cheong yes that is the limit
however
Typical maximum read sizes are often in the range of 16 KiB to 128 KiB. For example:
but response might be more for network I/O
let me double check
yea looks like a different situatoin for file probs
altho i still like having a loop just in case so can perhaps generalize over network stored CSVs
Google Chrome 2024-10-03 22.05.56.png
d
There's already a loop so it just processes whatever number of bytes the os returned. I think it makes sense to do some CPU work for each syscall to disk (or filesystem cache)
a
tbh i think it would need profiling
another issue
is that there might be a long string
that might cross boundaries if read is super small
d
Yeah 4mb was an arbitrary multiple of the page size anyway, definitely need to tune at some point but that's more of a todo
a
ok pushed my current progress not compiling yet but has my current progress
didnt rewrite historuy so we can always revert
but not planning on having to
d
awesome, thank you!
also I think your previous implementation did not read 4MiB into each slab as you just did one 
read
op
Fwiw I just tried printing the bytes read returned from the 
read()
and it's giving me 
4194304
bytes consistently. Between AI and myself, I trust myself more 🙂
a
yea look at second AI message @Desmond Cheong
think first message was just referring to net I/O operaitons
d
not sure if that's entirely correct for network i/o either. The 
read()
syscall over network should still go up to 2gb (for linux, posix limit might be smaller I believe). Packet sizes themselves might be smaller, with traditional tcp or udp the max packet size is 65536, but using other protocols this can always go to several gigabytes. Would have to fact check myself, but I think in this case the answer is kinda nuanced, and I wouldn't take the first (or even second) ai message too seriously
a
well my experience is often network will be smaller
from doing stuff with hyperion (my mc server)
w/ TCP
note L1 is 64 KB
Anyway fine keeping it single read
Either way is fine
Thinking for PR could be beneficial to pair program to cut down review time? Idk
d
note L1 is 64 KB
yeah but we read into memory first, then it gets pulled into cache as we operate on it
Thinking for PR could be beneficial to pair program to cut down review time? Idk
yeah down to pair
a
Often the TCP buffer sizes are small enough to fit in L1
At least on macOS from my experience
The default buffer size
Can get adjusted automatically by OS I think though
Again really only useful if we benchmark imo
Or test
In real world
4 Views
Open in Slack
PreviousNext
Powered by