https://www.getdaft.io logo
Join Slack
Powered by
Does anyone have any pointers for reading data fro...
# general
y
Does anyone have any pointers for reading data from s3-like filesystems that have custom ssl certs?
daft.exceptions.MiscTransientError: Misc Transient error trying to read [redacted] Details: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: hyper::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 2091 }]))) }, X509VerifyResult { code: 20, error: "unable to get local issuer certificate" })), connection: Unknown } })
I’ve tried setting SSL_CERT_FILE env var but doesn’t seem to work
j
I think @Sammy Sidhu would know more
s
Hi @Yuri Gorokhov, We use OpenSSL under the hood. So I would expect 
SSL_CERT_FILE
to work, but it's interesting it doesn't. Are you running Daft on a single node or distributed? In the mean time you can verify that everything else works by setting 
verify_ssl=False
in the S3Config https://www.getdaft.io/projects/docs/en/latest/api_docs/doc_gen/io_configs/daft.io.S3Config.html
y
Hi Sammy I’ve figured it out. SSL certs weren’t properly setup. Thank you!
I’m running into another issue though
It seems when using the pyarrow backend (instead of native) the io_config is not respected.
Is that expected? And if so, what would be the proper way to configure the pyarrow backend with custom s3 endpoints
s
For reads, I would recommend our native backend which will be much more performant and stable than the pyarrow backend (which is in the process of being deprecated) Is there a reason you are using pyarrow?
y
Using it for writes :)
I was just testing the read to confirm it wasn’t picking up the io_config. The native read works for me, but the write fails.
Does that also not respect the io_config?
s
It seems that all the options in 
io_config
are not forwarded here: https://github.com/Eventual-Inc/Daft/blob/ab1b772ab2efc3374d5de30c62e55202ee1b107b/daft/filesystem.py#L212 What options are you using? I can make a PR to translate the options
y
Copy code
io_config = IOConfig(
            s3=S3Config(
                endpoint_url='xxx',
                region_name="us-west-2",
                connect_timeout_ms=300_000,
                profile_name="xxx",
            )
        )
The problem is fixed when I set:
Copy code
export AWS_PROFILE=xxx
so I suspect the profile_name may be the culprit
s
Ah I see, yeah that fix makes sense!
It seems like Pyarrow S3FileSystem doesn't have a profile option https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html
So your workaround makes sense.
y
that makes sense.
Thanks a lot.
s
We're going to build our own native parquet writers soon. So this experience will be much better Thanks for bearing with us
y
Awesome, thank you! I have a workaround, all good 🙂
🔥 1
4 Views
Open in Slack
PreviousNext
Powered by