Qdrant Vector Database Creation

Qdrant Vector Database Creation#

import jsonlines
import os
import requests
import typing as t

from getpass import getpass
from langchain_huggingface import HuggingFaceEmbeddings
from langchain.schema import Document
from langchain_qdrant import Qdrant
from llama_cpp import Llama
import pandas as pd
from tqdm.notebook import tqdm
from qdrant_client import QdrantClient

Download Data#

from ssec_tutorials import ASTROPH_ARXIV_ABSTRACTS, ASTROPY_GITHUB, download_astroph_arxiv_abstracts, download_astropy_github_documents, fetch_and_process_github_rst_files, download_qdrant_data, QDRANT_COLLECTION_NAME, QDRANT_PATH
from ssec_tutorials.scipy_conf import load_docs_from_jsonl

download_astropy_github_documents()

PosixPath('/Users/a42/.cache/ssec_tutorials/astropy-github.jsonl')

assert os.path.exists(ASTROPY_GITHUB)

download_astroph_arxiv_abstracts()

astro-ph arXiv abstracts already exist at /Users/a42/.cache/ssec_tutorials/astro-ph-arXiv-abstracts.pkl

PosixPath('/Users/a42/.cache/ssec_tutorials/astro-ph-arXiv-abstracts.pkl')

assert os.path.exists(ASTROPH_ARXIV_ABSTRACTS)

# Download qdrant vector database
download_qdrant_data()

PosixPath('/Users/a42/.cache/ssec_tutorials/scipy_qdrant_100k')

GitHub Documents#

# Enter your GitHub Personal Access Token securely
ACCESS_TOKEN = getpass(prompt="GitHub Personal Access Token: ")

# Usage example
repository = 'astropy/astropy'
branch = 'main'
docs_path = 'docs'

# Load Astropy docs from ASTROPY_GITHUB if already downloaded
if not os.path.exists(ASTROPY_GITHUB):
    github_documents = fetch_and_process_github_rst_files(
        github_repo=repository, 
        github_branch=branch, 
        github_documents=docs_path,
        github_personal_access_token=ACCESS_TOKEN
    )
else:
    github_documents = load_docs_from_jsonl(ASTROPY_GITHUB)

len(github_documents)

311

Arxiv Abstracts#

# We will use the already pickled file but refer to the notebook in the Appendix if you are interested in understanding how we built it
astro_df = pd.read_pickle(ASTROPH_ARXIV_ABSTRACTS)

print("Number of astrophysics papers: ", len(astro_df))

Number of astrophysics papers:  338658

astro_df.head()
id title abstract
0 704.0009 The Spitzer c2d Survey of Large, Nearby, Inste... We discuss the results from the combined IRA...
1 704.0017 Spectroscopic Observations of the Intermediate... Results from spectroscopic observations of t...
2 704.0023 ALMA as the ideal probe of the solar chromosphere The very nature of the solar chromosphere, i...
3 704.0044 Astrophysical gyrokinetics: kinetic and fluid ... We present a theoretical framework for plasm...
4 704.0048 Inference on white dwarf binary systems using ... We report on the analysis of selected single... 
# Randomly sample n rows to reduce the eventual vector database size.
astro_df = astro_df.sample(n=100000)

Documents Loader#

LangChain helps load different documents (.txt, .pdf, .docx, .csv, .xlsx, .json) to feed into the LLM. The Document Loader even allows YouTube audio parsing and loading as part of unstructured document loading.

Once loaded into the LangChain, the document can be pre-processed in different ways as required in the LLM application.

from langchain_community.document_loaders import DataFrameLoader

# Load the dataframe full of abstracts
# to memory in the form of LangChain Document objects
loader = DataFrameLoader(astro_df, page_content_column="abstract") 
astrophysics_abstracts_documents = loader.load()

print("Number of astrophysics papers: ", len(astrophysics_abstracts_documents))

Number of astrophysics papers:  100000

all_documents = astrophysics_abstracts_documents + github_documents
print("Total Number of Documents: ", len(all_documents))

Total Number of Documents:  100311

Qdrant Creation#

QDRANT_PATH

PosixPath('/Users/a42/.cache/ssec_tutorials/scipy_qdrant')

QDRANT_COLLECTION_NAME

'arxiv_astro-ph_abstracts_astropy_github_documentation'

model = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L12-v2")

/Users/a42/anaconda3/envs/ssec-scipy2024/lib/python3.11/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py:11: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  from tqdm.autonotebook import tqdm, trange
/Users/a42/anaconda3/envs/ssec-scipy2024/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(

if QDRANT_PATH.exists():
    print(f"Qdrant Vector Database Collection already exists in {QDRANT_PATH}, load it")
    client = QdrantClient(path=str(QDRANT_PATH))
    qdrant = Qdrant(
        client=client,
        collection_name=QDRANT_COLLECTION_NAME,
        embeddings=model
    )
else:
    print(f"Creating new Qdrant collection '{QDRANT_COLLECTION_NAME}' from {len(all_documents)} documents")
    
    # Load the documents into a Qdrant Vector Database Collection
    # this will save locally in the qdrant_path as sqlite
    qdrant = Qdrant.from_documents(
        documents=all_documents,
        embedding=model,
        path=str(QDRANT_PATH),
        collection_name=QDRANT_COLLECTION_NAME,
    )

Qdrant Vector Database Collection already exists in /Users/a42/.cache/ssec_tutorials/scipy_qdrant, load it

# Number of documents in qdrant
qdrant.client.count(collection_name=QDRANT_COLLECTION_NAME)

CountResult(count=100311)

# Setup the retriever for later step
retriever = qdrant.as_retriever(search_type="mmr", search_kwargs={"k": 2})

retriever.invoke("What is dark matter?")

[Document(page_content='  One of the great scientific enigmas still unsolved, the existence of dark\nmatter, is reviewed. Simple gravitational arguments imply that most of the mass\nin the Universe, at least 90%, is some (unknown) non-luminous matter. Some\nparticle candidates for dark matter are discussed with particular emphasis on\nthe neutralino, a particle predicted by the supersymmetric extension of the\nStandard Model of particle physics. Experiments searching for these relic\nparticles, carried out by many groups around the world, are also discussed.\nThese experiments are becoming more sensitive every year and in fact one of the\ncollaborations claims that the first direct evidence for dark matter has\nalready been observed.\n', metadata={'id': 'hep-ph/0110122', 'title': 'The Enigma of the Dark Matter', '_id': '4ab99f7c922747d9a6a34b855d959779', '_collection_name': 'arxiv_astro-ph_abstracts_astropy_github_documentation'}),
 Document(page_content='  Dark matter could be composed of compact dark objects (CDOs). These objects\nmay interact very weakly with normal matter and could move freely {\\it inside}\nthe Earth. A CDO moving in the inner core of the Earth will have an orbital\nperiod near 55 min and produce a time dependent signal in a gravimeter. Data\nfrom superconducting gravimeters rule out such objects moving inside the Earth\nunless their mass $m_D$ and or orbital radius $a$ are very small so that $m_D\\,\na < 1.2\\times 10^{-13}M_\\oplus R_\\oplus$. Here $M_\\oplus$ and $R_\\oplus$ are\nthe mass and radius of the Earth.\n', metadata={'id': 1912.0094, 'title': 'Gravimeter search for compact dark matter objects moving in the Earth', '_id': '97fa0dcbd2aa45d28dfccaa150e724e2', '_collection_name': 'arxiv_astro-ph_abstracts_astropy_github_documentation'})]

retriever.invoke("How can I perform celestial coordinate transformations?")

[Document(page_content='.. _astropy-coordinates-transforming:\n\nTransforming between Systems\n****************************\n\n`astropy.coordinates` supports a rich system for transforming\ncoordinates from one frame to another. While common astronomy frames\nare built into Astropy, the transformation infrastructure is dynamic.\nThis means it allows users to define new coordinate frames and their\ntransformations. The topic of writing your own coordinate frame or\ntransforms is detailed in :ref:`astropy-coordinates-design`, and this\nsection is focused on how to *use* transformations.\n\nThe full list of built-in coordinate frames, the included transformations,\nand the frame names are shown as a (clickable) graph in the\n`~astropy.coordinates` API documentation.\n\nExamples\n--------\n\n..\n  EXAMPLE START\n  Transforming Coordinates to Another Frame\n\nThe recommended method of transformation is shown below::\n\n    >>> import astropy.units as u\n    >>> from astropy.coordinates import SkyCoord\n    >>> gc = SkyCoord(l=0*u.degree, b=45*u.degree, frame=\'galactic\')\n    >>> gc.fk5  # doctest: +FLOAT_CMP\n    <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg\n        ( 229.27251463, -1.12844288)>\n\nWhile this appears to be ordinary attribute-style access, it is actually\nsyntactic sugar for the more general\n:meth:`~astropy.coordinates.SkyCoord.transform_to` method, which can\naccept either a frame name, class, or instance::\n\n    >>> from astropy.coordinates import FK5\n    >>> gc.transform_to(\'fk5\')  # doctest: +FLOAT_CMP\n    <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg\n        ( 229.27251463, -1.12844288)>\n    >>> gc.transform_to(FK5)  # doctest: +FLOAT_CMP\n    <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg\n        ( 229.27251463, -1.12844288)>\n    >>> gc.transform_to(FK5(equinox=\'J1980.0\'))  # doctest: +FLOAT_CMP\n    <SkyCoord (FK5: equinox=J1980.000): (ra, dec) in deg\n        ( 229.0146935, -1.05560349)>\n\n..\n  EXAMPLE END\n\n..\n  EXAMPLE START\n  Using SkyCoord Objects as the Frame in Transformations\n\nAs a convenience, it is also possible to use a |SkyCoord| object as the frame in\n:meth:`~astropy.coordinates.SkyCoord.transform_to`. This allows for putting one\ncoordinate object into the frame of another::\n\n    >>> sc = SkyCoord(ra=1.0, dec=2.0, unit=\'deg\', frame=FK5, equinox=\'J1980.0\')\n    >>> gc.transform_to(sc)  # doctest: +FLOAT_CMP\n    <SkyCoord (FK5: equinox=J1980.000): (ra, dec) in deg\n        ( 229.0146935, -1.05560349)>\n\n..\n  EXAMPLE END\n\n..\n  EXAMPLE START\n  Self Transformations of Coordinate Frames\n\nSome coordinate frames (including `~astropy.coordinates.FK5`,\n`~astropy.coordinates.FK4`, and `~astropy.coordinates.FK4NoETerms`) support\n"self transformations," meaning the *type* of frame does not change, but the\nframe attributes do. An example is precessing a coordinate from one equinox\nto another in an equatorial frame. This is done by passing ``transform_to`` a\nframe class with the relevant attributes, as shown below. Note that these\nframes use a default equinox if you do not specify one::\n\n    >>> fk5c = SkyCoord(\'02h31m49.09s\', \'+89d15m50.8s\', frame=FK5)\n    >>> fk5c.equinox\n    <Time object: scale=\'tt\' format=\'jyear_str\' value=J2000.000>\n    >>> fk5c  # doctest: +FLOAT_CMP\n    <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg\n        ( 37.95454167,  89.26411111)>\n    >>> fk5_2005 = FK5(equinox=\'J2005\')  # String initializes an astropy.time.Time object\n    >>> fk5c.transform_to(fk5_2005)  # doctest: +FLOAT_CMP\n    <SkyCoord (FK5: equinox=J2005.000): (ra, dec) in deg\n        ( 39.39317639,  89.28584422)>\n\nYou can also specify the equinox when you create a coordinate using a\n`~astropy.time.Time` object::\n\n    >>> from astropy.time import Time\n    >>> fk5c = SkyCoord(\'02h31m49.09s\', \'+89d15m50.8s\',\n    ...                 frame=FK5(equinox=Time(\'J1970\')))\n    >>> fk5_2000 = FK5(equinox=Time(2000, format=\'jyear\'))\n    >>> fk5c.transform_to(fk5_2000)  # doctest: +FLOAT_CMP\n    <SkyCoord (FK5: equinox=2000.0): (ra, dec) in deg\n        ( 48.023171,  89.38672485)>\n\nThe same lower-level frame classes also have a\n:meth:`~astropy.coordinates.BaseCoordinateFrame.transform_to` method\nthat works the same as above, but they do not support attribute-style\naccess. They are also subtly different in that they only use frame\nattributes present in the initial or final frame, while |SkyCoord|\nobjects use any frame attributes they have for all transformation\nsteps. So |SkyCoord| can always transform from one frame to another and\nback again without change, while low-level classes may lose information\nand hence often do not round-trip.\n\n..\n  EXAMPLE END\n\n.. _astropy-coordinates-transforming-ephemerides:\n\nTransformations and Solar System Ephemerides\n============================================\n\nSome transformations (e.g., the transformation between\n`~astropy.coordinates.ICRS` and `~astropy.coordinates.GCRS`) require the use of\na Solar System ephemeris to calculate the position and velocity of the Earth\nand Sun. By default, transformations are calculated using built-in\n`ERFA <https://github.com/liberfa/erfa>`_ routines, but they can also use more\nprecise ones using the JPL ephemerides (which are derived from dynamical\nmodels).\n\nExample\n-------\n\n..\n  EXAMPLE START\n  Calculating Transformations Using Solar System Ephemeris\n\nTo use the JPL ephemerides, use the\n`~astropy.coordinates.solar_system_ephemeris` context manager, as shown below:\n\n.. doctest-requires:: jplephem\n\n    >>> from astropy.coordinates import solar_system_ephemeris\n    >>> from astropy.coordinates import GCRS\n    >>> with solar_system_ephemeris.set(\'jpl\'): # doctest: +REMOTE_DATA +IGNORE_OUTPUT\n    ...     fk5c.transform_to(GCRS(obstime=Time("J2000"))) # doctest: +REMOTE_DATA +IGNORE_OUTPUT\n\nFor locations at large distances from the Solar system, using the JPL\nephemerides will make a negligible difference on the order of micro-arcseconds.\nFor nearby objects, such as the Moon, the difference can be of the\norder of milli-arcseconds. For more details about what ephemerides\nare available, including the requirements for using JPL ephemerides, see\n:ref:`astropy-coordinates-solarsystem`.\n\n..\n  EXAMPLE END\n', metadata={'title': 'Transforming', 'url': 'https://raw.githubusercontent.com/astropy/astropy/main/docs/coordinates/transforming.rst', '_id': '7591267e1b4e423496b3a417848de92d', '_collection_name': 'arxiv_astro-ph_abstracts_astropy_github_documentation'}),
 Document(page_content="  Gorski et al (1999b) have earlier presented the outline of a\npixelisation-to-spherical-coordinate transformation scheme which simultaneously\nsatisfies three properties which are especially useful for rapid analyses of\nmaps on a sphere: (i) equal spacing of pixels along lines of constant latitude,\n(ii) equal pixel `areas' (solid angles) and (iii) hierarchical scaling with\nincreasing numbers of pixels. Their outline is based on the division of the\nsphere into twelve regions covering equal solid angles, which are\nhierarchically subdivided in a way compatible with these three criteria. In\nthis paper, a complete derivation of this scheme is presented, including, in\nparticular, (1) the angle theta^* defining the limit between polar and\nequatorial regions, and (2) the transformations from the unit interval [0,1]\n\\wedge [0,1] to spherical coordinates in a polar region.\n", metadata={'id': 'astro-ph/0409533', 'title': 'A Solution to the Isolatitude, Equi-area, Hierarchical Pixel-Coordinate\n  System', '_id': '98b95594b76a4e9eaa6f0d95c167d500', '_collection_name': 'arxiv_astro-ph_abstracts_astropy_github_documentation'})]

# Post-processing
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

print(format_docs(retriever.invoke("How can I perform celestial coordinate transformations?")))

.. _astropy-coordinates-transforming:

Transforming between Systems
****************************

`astropy.coordinates` supports a rich system for transforming
coordinates from one frame to another. While common astronomy frames
are built into Astropy, the transformation infrastructure is dynamic.
This means it allows users to define new coordinate frames and their
transformations. The topic of writing your own coordinate frame or
transforms is detailed in :ref:`astropy-coordinates-design`, and this
section is focused on how to *use* transformations.

The full list of built-in coordinate frames, the included transformations,
and the frame names are shown as a (clickable) graph in the
`~astropy.coordinates` API documentation.

Examples
--------

..
  EXAMPLE START
  Transforming Coordinates to Another Frame

The recommended method of transformation is shown below::

    >>> import astropy.units as u
    >>> from astropy.coordinates import SkyCoord
    >>> gc = SkyCoord(l=0*u.degree, b=45*u.degree, frame='galactic')
    >>> gc.fk5  # doctest: +FLOAT_CMP
    <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg
        ( 229.27251463, -1.12844288)>

While this appears to be ordinary attribute-style access, it is actually
syntactic sugar for the more general
:meth:`~astropy.coordinates.SkyCoord.transform_to` method, which can
accept either a frame name, class, or instance::

    >>> from astropy.coordinates import FK5
    >>> gc.transform_to('fk5')  # doctest: +FLOAT_CMP
    <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg
        ( 229.27251463, -1.12844288)>
    >>> gc.transform_to(FK5)  # doctest: +FLOAT_CMP
    <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg
        ( 229.27251463, -1.12844288)>
    >>> gc.transform_to(FK5(equinox='J1980.0'))  # doctest: +FLOAT_CMP
    <SkyCoord (FK5: equinox=J1980.000): (ra, dec) in deg
        ( 229.0146935, -1.05560349)>

..
  EXAMPLE END

..
  EXAMPLE START
  Using SkyCoord Objects as the Frame in Transformations

As a convenience, it is also possible to use a |SkyCoord| object as the frame in
:meth:`~astropy.coordinates.SkyCoord.transform_to`. This allows for putting one
coordinate object into the frame of another::

    >>> sc = SkyCoord(ra=1.0, dec=2.0, unit='deg', frame=FK5, equinox='J1980.0')
    >>> gc.transform_to(sc)  # doctest: +FLOAT_CMP
    <SkyCoord (FK5: equinox=J1980.000): (ra, dec) in deg
        ( 229.0146935, -1.05560349)>

..
  EXAMPLE END

..
  EXAMPLE START
  Self Transformations of Coordinate Frames

Some coordinate frames (including `~astropy.coordinates.FK5`,
`~astropy.coordinates.FK4`, and `~astropy.coordinates.FK4NoETerms`) support
"self transformations," meaning the *type* of frame does not change, but the
frame attributes do. An example is precessing a coordinate from one equinox
to another in an equatorial frame. This is done by passing ``transform_to`` a
frame class with the relevant attributes, as shown below. Note that these
frames use a default equinox if you do not specify one::

    >>> fk5c = SkyCoord('02h31m49.09s', '+89d15m50.8s', frame=FK5)
    >>> fk5c.equinox
    <Time object: scale='tt' format='jyear_str' value=J2000.000>
    >>> fk5c  # doctest: +FLOAT_CMP
    <SkyCoord (FK5: equinox=J2000.000): (ra, dec) in deg
        ( 37.95454167,  89.26411111)>
    >>> fk5_2005 = FK5(equinox='J2005')  # String initializes an astropy.time.Time object
    >>> fk5c.transform_to(fk5_2005)  # doctest: +FLOAT_CMP
    <SkyCoord (FK5: equinox=J2005.000): (ra, dec) in deg
        ( 39.39317639,  89.28584422)>

You can also specify the equinox when you create a coordinate using a
`~astropy.time.Time` object::

    >>> from astropy.time import Time
    >>> fk5c = SkyCoord('02h31m49.09s', '+89d15m50.8s',
    ...                 frame=FK5(equinox=Time('J1970')))
    >>> fk5_2000 = FK5(equinox=Time(2000, format='jyear'))
    >>> fk5c.transform_to(fk5_2000)  # doctest: +FLOAT_CMP
    <SkyCoord (FK5: equinox=2000.0): (ra, dec) in deg
        ( 48.023171,  89.38672485)>

The same lower-level frame classes also have a
:meth:`~astropy.coordinates.BaseCoordinateFrame.transform_to` method
that works the same as above, but they do not support attribute-style
access. They are also subtly different in that they only use frame
attributes present in the initial or final frame, while |SkyCoord|
objects use any frame attributes they have for all transformation
steps. So |SkyCoord| can always transform from one frame to another and
back again without change, while low-level classes may lose information
and hence often do not round-trip.

..
  EXAMPLE END

.. _astropy-coordinates-transforming-ephemerides:

Transformations and Solar System Ephemerides
============================================

Some transformations (e.g., the transformation between
`~astropy.coordinates.ICRS` and `~astropy.coordinates.GCRS`) require the use of
a Solar System ephemeris to calculate the position and velocity of the Earth
and Sun. By default, transformations are calculated using built-in
`ERFA <https://github.com/liberfa/erfa>`_ routines, but they can also use more
precise ones using the JPL ephemerides (which are derived from dynamical
models).

Example
-------

..
  EXAMPLE START
  Calculating Transformations Using Solar System Ephemeris

To use the JPL ephemerides, use the
`~astropy.coordinates.solar_system_ephemeris` context manager, as shown below:

.. doctest-requires:: jplephem

    >>> from astropy.coordinates import solar_system_ephemeris
    >>> from astropy.coordinates import GCRS
    >>> with solar_system_ephemeris.set('jpl'): # doctest: +REMOTE_DATA +IGNORE_OUTPUT
    ...     fk5c.transform_to(GCRS(obstime=Time("J2000"))) # doctest: +REMOTE_DATA +IGNORE_OUTPUT

For locations at large distances from the Solar system, using the JPL
ephemerides will make a negligible difference on the order of micro-arcseconds.
For nearby objects, such as the Moon, the difference can be of the
order of milli-arcseconds. For more details about what ephemerides
are available, including the requirements for using JPL ephemerides, see
:ref:`astropy-coordinates-solarsystem`.

..
  EXAMPLE END


  AIMS: An alternative to the traditional method for modeling kinematics of the
Earth's rotation is proposed. The purpose of developing the new approach is to
provide a self-consistent and simple description of the Earth's rotation in a
way that can be estimated directly from observations without using intermediate
quantities.
  METHODS: Instead of estimating the time series of pole coordinates, the
UT1--TAI angles, their rates, and the daily offsets of nutation, it is proposed
to estimate coefficients of the expansion of a small perturbational rotation
vector into basis functions. The resulting transformation from the terrestrial
coordinate system to the celestial coordinate system is formulated as a product
of an a priori matrix of a finite rotation and an empirical vector of a
residual perturbational rotation. In the framework of this approach, the
specific choice of the a priori matrix is irrelevant, provided the angles of
the residual rotation are small enough to neglect their squares. The
coefficients of the expansion into the B-spline and Fourier bases, together
with estimates of other nuisance parameters, are evaluated directly from
observations of time delay or time range in a single least square solution.
  RESULTS: This approach was successfully implemented in a computer program for
processing VLBI observations. The dataset from 1984 through 2006 was analyzed.
The new procedure adequately represents the Earth's rotation, including slowly
varying changes in UT1--TAI and polar motion, the forced nutations, the free
core nutation, and the high frequency variations of polar motion and UT1.
Contents