IZj0ddlmZddlmZmZmZmZddlZddl Z ddl Z ddl m Z mZeeede jee jfZeeeje je jfZeeefZdZGdd eZd ed efd Zd ee jde jd ee jfdZ ddedee jd e j fdZ!de jfdZ"defdZ#dS))Path)IterableListOptionalUnionN)_check_for_pandaspandasz pd.DataFramevectorc*eZdZdZdefdZdefdZdS) CredentialzCredential fieldreturncdSNz********selfs Z/Users/jameslopez/projects/MentorCore/.venv/lib/python3.11/site-packages/lancedb/common.py__repr__zCredential.__repr__zcdSrrrs r__str__zCredential.__str__rrN)__name__ __module__ __qualname____doc__strrrrrrr r sP#rr urirc t|SN)r)rs r sanitize_urir"s s88Or input_iterschemac#K|D]}t|tjstdt ||j|kr tj|g| d}n2#tj j $rtd|d|jwxYw|VdS)a Wrapper around an iterator of record batches. If the batches don't match the schema, try to cast them to the schema. If that fails, raise an error. This is helpful for users who might have written the iterator with default data types in PyArrow, but specified more specific types in the schema. For example, PyArrow defaults to float64 for floating point types, but Lance uses float32 for vectors. zExpected RecordBatch, got rzjInput RecordBatch iterator yielded a batch with schema that does not match the expected schema. Expected: z Got: N) isinstancepa RecordBatch TypeErrortyper$Table from_batchescast to_batcheslib ArrowInvalid ValueError)r#r$batchs r_casting_recordbatch_iterr3"s  %00 HFeFFGG G <6 ! ! --ug66;;FCCNNPPQRS6&    ,GM,,"\,,     s A B/Cdatact|rMt|tjr3tj||St|tjr|St|tjr2tj |gSt|tj j r6tj j |St|tj j r|St|tjr|St|jdr6|jjdkr&|St|t*rA|0t-||}tj ||St/dt1dt|d)z7Convert various types of input into a RecordBatchReaderr$polars DataFrameNz>Must provide schema to write dataset from RecordBatch iterablezUnknown data type z. Supported types: list of dicts, pandas DataFrame, polars DataFrame, pyarrow Table/RecordBatch, or Pydantic models. See https://docs.lancedb.com/tables/ for examples.)r r&pdr8r'r+ from_pandas to_readerr(r,datasetDatasetScanner from_datasetRecordBatchReaderr*r startswith __class__rto_arrowrr3r1r))r4r$s rdata_to_readerrD>s" :dBL#A#A" x##D#88BBDDD D"( # #  ~~ D". ) ) x$$dV,,66888 D"*, - - z!..t44>>@@@ D"*, - - ~~ D". / /   T ((22 N #{ 2 2}}((*** D( # #  ,T6::D'44VTBB BP  Ad A A A   rc@|jt|jdSdS)z. Make sure the metadata is valid utf8 N)metadata_validate_metadatar6s rvalidate_schemarHgs*"6?+++++#"rrFc|D]v\}}t|tr8 |d1#t$rt d|dwxYwt|t rt|wdS)zo Make sure the metadata values are valid utf8 (can be nested) Raises ValueError if not valid utf8 utf8z Metadata key zG is not valid utf8. Consider base64 encode for generic binary metadata.N)itemsr&bytesdecodeUnicodeDecodeErrorr1dictrG)rFkvs rrGrGos   " "1 a   "     %    JAJJJ  4  " q ! ! ! " "s AA$r!)$pathlibrtypingrrrrnumpynppyarrowr'pyarrow.dataset dependenciesr r r9rOr+r(DATAlistndarrayArray ChunkedArrayVECrURIVECTOR_COLUMN_NAMEr r"Schemar3r@rDrHrGrrrrbs 22222222222299999999 T$Z8BN3K KL D"*bh 78 CIcc(24) bn:/3& & &  +& & & & & R,BI,,,,"""""""r