RE: [Patch v5 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob for Azure VM

[Long Li]

> Subject: RE: [Patch v5 0/3] Introduce a driver to support host accelerated access
> to Microsoft Azure Blob for Azure VM
> 
> > Subject: Re: [Patch v5 0/3] Introduce a driver to support host
> > accelerated access to Microsoft Azure Blob for Azure VM
> >
> > On Sat, Aug 07, 2021 at 06:29:06PM +0000, Long Li wrote:
> > > > I still think this "model" is totally broken and wrong overall.
> > > > Again, you are creating a custom "block" layer with a character
> > > > device, forcing all userspace programs to use a custom library
> > > > (where is it
> > at?) just to get their data.
> > >
> > > The Azure Blob library (with source code) is available in the
> > > following
> > languages:
> > > Java:
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2FAzure%2Fazure-sdk-for-
> > java%2Ftree%2Fmain%2Fsdk%2Fstorage%2Faz
> > > ure-storage-
> > blob&data=04%7C01%7Clongli%40microsoft.com%7C778083147
> > >
> > 8ed49b16e6308d95a2b7ae8%7C72f988bf86f141af91ab2d7cd011db47%7C1
> > %7C0%7C6
> > >
> > 37639965101378114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
> > DAiLCJQIjoi
> > >
> > V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wcNhsEoH
> > LV0VBc
> > > uDf0CVXl7W0Ug9Cj7Q92%2Bw6qizroU%3D&reserved=0
> > > JavaScript:
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2FAzure%2Fazure-sdk-for-
> > js%2Ftree%2Fmain%2Fsdk%2Fstorage%2Fstor
> > > age-
> > blob&data=04%7C01%7Clongli%40microsoft.com%7C7780831478ed49b
> > 16
> > >
> > e6308d95a2b7ae8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C
> > 637639965
> > >
> > 101378114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo
> > iV2luMzIi
> > >
> > LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=I%2FfhdPX3Unz6S3
> > eBPcpl
> > > %2Bh55nKoV0u%2FO0%2BYgjLy4grQ%3D&reserved=0
> > > Python:
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2FAzure%2Fazure-sdk-for-
> > python%2Ftree%2Fmain%2Fsdk%2Fstorage%2F
> > > azure-storage-
> > blob&data=04%7C01%7Clongli%40microsoft.com%7C7780831
> > >
> > 478ed49b16e6308d95a2b7ae8%7C72f988bf86f141af91ab2d7cd011db47%7
> > C1%7C0%7
> > >
> > C637639965101378114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> > MDAiLCJQIj
> > >
> > oiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aAwsi%2
> > BPVsN
> > > tsDMJ7rKnRDigNc41fIao031lde247Nc0%3D&reserved=0
> > > Go:
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2FAzure%2Fazure-storage-blob-
> > go&data=04%7C01%7Clongli%40mic
> > >
> > rosoft.com%7C7780831478ed49b16e6308d95a2b7ae8%7C72f988bf86f141a
> > f91ab2d
> > >
> > 7cd011db47%7C1%7C0%7C637639965101378114%7CUnknown%7CTWFpbG
> > Zsb3d8eyJWIj
> > >
> > oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C10
> > 00&am
> > >
> > p;sdata=43JhbGsYQxA%2FoivNd7C3z7DSYO%2FPONCoaW2v7TN6xEU%3D&a
> > mp;reserve
> > > d=0
> > > .NET:
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2FAzure%2Fazure-sdk-for-
> > net%2Ftree%2Fmain%2Fsdk%2Fstorage%2FAzu
> > >
> > re.Storage.Blobs&data=04%7C01%7Clongli%40microsoft.com%7C77808
> > 3147
> > >
> > 8ed49b16e6308d95a2b7ae8%7C72f988bf86f141af91ab2d7cd011db47%7C1
> > %7C0%7C6
> > >
> > 37639965101378114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
> > DAiLCJQIjoi
> > >
> > V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6ClMeURlt
> > cBv1q
> > > 7l7PGGrxXVJbVDt9uMBlwoIVh7Wpw%3D&reserved=0
> > > PHP:
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2FAzure%2Fazure-storage-php%2Ftree%2Fmaster%2Fazure-
> > storage-blo
> > >
> > b&data=04%7C01%7Clongli%40microsoft.com%7C7780831478ed49b16
> > e6308d9
> > >
> > 5a2b7ae8%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6376399
> > 651013781
> > >
> > 14%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
> > LCJBTiI
> > >
> > 6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DuZO539vd76c%2Byaqjn
> > hetp%2B3T
> > > i0b74601ZkNe39SNK4%3D&reserved=0
> > > Ruby:
> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > th
> > > ub.com%2Fazure%2Fazure-storage-
> > ruby%2Ftree%2Fmaster%2Fblob&data=04
> > > %7C01%7Clongli%40microsoft.com%7C7780831478ed49b16e6308d95a2b
> > 7ae8%7C72
> > >
> > f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637639965101378114%7
> > CUnknown%
> > >
> > 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwi
> > LCJX
> > >
> > VCI6Mn0%3D%7C1000&sdata=6Zviu1IuRQE2do9bDCae2iJv0W2KOJu90t
> > XSR6kDAR
> > > 4%3D&reserved=0
> > > C++:
> > >
> > C++https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fg
> > > C++ithub.com%2FAzure%2Fazure-sdk-for-
> > cpp%2Ftree%2Fmain%2Fsdk%2Fstorage
> > > C++%23azure-storage-client-library-for-
> > c&data=04%7C01%7Clongli%40m
> > >
> > C++icrosoft.com%7C7780831478ed49b16e6308d95a2b7ae8%7C72f988bf86
> > f141af9
> > >
> > C++1ab2d7cd011db47%7C1%7C0%7C637639965101388074%7CUnknown%
> > 7CTWFpbGZsb3
> > >
> >
> C++d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> > 0%3
> > >
> > C++D%7C1000&sdata=HH6jrqREWQ%2BkoRR%2Fsb02wRXnuLU5il4Erzm
> > rBvUZu5w%
> > > C++3D&reserved=0
> >
> > And why wasn't this linked to in the changelog here?
> >
> > In looking at the C code above, where is the interaction with this Linux driver?
> > I can't seem to find it...

Greg,

I apologize for the delay. I have attached the Java transport library (a tgz file) in the email. The file is released for review under "The MIT License (MIT)".

The transport library implemented functions needed for reading from a Block Blob using this driver. The function for transporting I/O is Java_com_azure_storage_fastpath_driver_FastpathDriver_read(), defined in "./src/fastpath/jni/fpjar_endpoint.cpp".

In particular, requestParams is in JSON format (REST) that is passed from a Blob application using Blob API for reading from a Block Blob.

For an example of how a Blob application using the transport library, please see Blob support for Hadoop ABFS:
https://github.com/apache/hadoop/pull/3309/commits/be7d12662e23a13e6cf10cf1fa5e7eb109738e7d

In ABFS, the entry point for using Blob I/O is at AbfsRestOperation executeRead() in hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java, from line 553 to 564, this function eventually calls into executeFastpathRead() in hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java.

ReadRequestParameters is the data that is passed to requestParams (described above) in the transport library. In this Blob application use-case, ReadRequestParameters has eTag and sessionInfo (sessionToken). They are both defined in this commit, and are treated as strings passed in JSON format to I/O issuing function Java_com_azure_storage_fastpath_driver_FastpathDriver_read() in the transport library using this driver.

Thanks,
Long

> 
> Those are existing Blob client libraries. The new code using this driver is being
> tested and has not been released to github.
> 
> I'm sorry I misunderstood your request. I'm asking the team to share the new
> code for review. I will send the code location for review soon.
> 
> Thanks,
> Long

Attachment: shared_library_for_azure_storage_fastpath_connector_for_java.tgz
Description: shared_library_for_azure_storage_fastpath_connector_for_java.tgz