{"id":23518,"date":"2023-10-17T11:34:42","date_gmt":"2023-10-17T09:34:42","guid":{"rendered":"https:\/\/info.gwdg.de\/news\/?p=23518"},"modified":"2023-10-17T11:48:27","modified_gmt":"2023-10-17T09:48:27","slug":"parallel-3d-point-cloud-data-analysis-with-dask","status":"publish","type":"post","link":"https:\/\/info.gwdg.de\/news\/parallel-3d-point-cloud-data-analysis-with-dask\/","title":{"rendered":"Parallel 3D Point Cloud Data analysis with Dask"},"content":{"rendered":"

There are many python packages that are tightly integrated with Dask which enables parallel data processing. For instance, consider xarray<\/code> package. This package is used to read datasets in netcdf, hdf5, zarr file formats. Dask comes in play when data is read from multile files where each file is treated as a chunk or by explictly specifying chunks<\/code> parameter to describe how the data needs to be chunked. laspy<\/code> python package is used for reading 3D point cloud data but unfortunately it lacks integration with Dask but still it is possible to use it when data is read from multiple files. Although this techique of loading data from multiple files is quite well known, having the chunks<\/code> parameter feature adds a great deal of flexibility. In this article, a demonstration of such a feature is presented.<\/p>\n

A more permanent version of this article with updates and fixes can be found in our Gitlab repository<\/a>.<\/p>\n

3D Point Cloud format – Las\/Laz<\/h2>\n

A brief overview of the las format. The 3D point cloud data is stored as a las file. As shown in the figure below the las file consists of header, variable length records, point cloud records and extended variable length records. Depending on the version of the las format specification, few details may vary in fields stored in point cloud records and may not have extended variable length records.<\/p>\n

\"https:\/\/pad.gwdg.de\/uploads\/9738de9a-6347-417d-9468-894bd2f1a859.png\"<\/p>\n

The header part has the metadata about the file contents like format version, number of point cloud records, size of each record in bytes, offset byte where the point cloud record starts, data-type of fields in the record, minimum and maximum values of the spatial co-ordinates, scaling factor, etc.,<\/p>\n

laspy<\/code> package provides convenience functions not just to read point cloud records but also allows to investigate metadata. Perhaps one interesting thing to figure out from metadata is memory requirements to load a las file. Here is a small utility function called info<\/code> to provide this information.<\/p>\n

import<\/span> humanize<\/span><\/span>\r\nimport<\/span> laspy<\/span><\/span>\r\nimport<\/span> os<\/span><\/span>\r\n<\/span>\r\ndef<\/span> info<\/span>(<\/span>filename<\/span>):<\/span><\/span>\r\n    naturalsize<\/span> =<\/span> lambda<\/span> x<\/span>:<\/span> humanize<\/span>.<\/span>naturalsize<\/span>(<\/span>x<\/span>,<\/span> binary<\/span>=<\/span>True<\/span>)<\/span><\/span>\r\n    with<\/span> open<\/span>(<\/span>filename<\/span>,<\/span> \"<\/span>rb<\/span>\"<\/span>)<\/span> as<\/span> fid<\/span>:<\/span><\/span>\r\n        reader<\/span> =<\/span> laspy<\/span>.<\/span>LasReader<\/span>(<\/span>fid<\/span>)<\/span><\/span>\r\n        npoints<\/span> =<\/span> reader<\/span>.<\/span>header<\/span>.<\/span>point_count<\/span><\/span>\r\n        point_size<\/span> =<\/span> reader<\/span>.<\/span>header<\/span>.<\/span>point_format<\/span>.<\/span>size<\/span><\/span>\r\n        memory_size<\/span> =<\/span> npoints<\/span> *<\/span> point_size<\/span><\/span>\r\n    print<\/span>(<\/span>f<\/span>\"<\/span>Filesize on disk: <\/span>{<\/span>naturalsize<\/span>(<\/span>os<\/span>.<\/span>path<\/span>.<\/span>getsize<\/span>(<\/span>filename<\/span>))<\/span>}<\/span>\"<\/span>)<\/span><\/span>\r\n    print<\/span>(<\/span>f<\/span>\"<\/span>Is data compressed: <\/span>{<\/span>reader<\/span>.<\/span>header<\/span>.<\/span>are_points_compressed<\/span>}<\/span>\"<\/span>)<\/span><\/span>\r\n    print<\/span>(<\/span>f<\/span>\"<\/span>records (points): <\/span>{<\/span>npoints<\/span>}<\/span> (<\/span>{<\/span>humanize<\/span>.<\/span>intword<\/span>(<\/span>npoints<\/span>)<\/span>}<\/span>)<\/span>\"<\/span>)<\/span><\/span>\r\n    print<\/span>(<\/span>f<\/span>\"<\/span>Each record size: <\/span>{<\/span>point_size<\/span>}<\/span> (<\/span>{<\/span>naturalsize<\/span>(<\/span>point_size<\/span>)<\/span>}<\/span>)<\/span>\"<\/span>)<\/span><\/span>\r\n    print<\/span>(<\/span>f<\/span>\"<\/span>Memory required: <\/span>{<\/span>memory_size<\/span>}<\/span> (<\/span>{<\/span>naturalsize<\/span>(<\/span>memory_size<\/span>)<\/span>}<\/span>)<\/span>\"<\/span>)<\/span><\/span>\r\n<\/pre>\n
Using the info<\/code> function to determine the memory requirements of a las file called densel-20230906-103153.laz<\/code><\/p>\n
filename<\/span> =<\/span> \"<\/span>.\/dense1-20230906-103153.laz<\/span>\"<\/span><\/span>\r\ninfo<\/span>(<\/span>filename<\/span>)<\/span><\/span>\r\n<\/pre>\n
Filesize on disk: 2.1 GiB<\/span>\r\nIs data compressed: True<\/span>\r\nrecords (points): 148919876 (148.9 million)<\/span>\r\nEach record size: 54 (54 Bytes)<\/span>\r\nMemory required: 8041673304 (7.5 GiB)<\/span>\r\n\r\n\r\n<\/pre>\n
Although this file occupies just 2.1 GiB in disk, the uncompressed data in memory requires at least 4 times more space in memory (7.5 GiB). This function is quite useful in planning compute node resource requirements. \"Each record size\"<\/code> indicates the number of bytes required to store the compressed data on disk.<\/p>\n
As mentioned in the introduction, the idea is not to physically split the file into multiple smaller files but logically divide the single file into multiple chunks so Dask can work with these chunks. The immediate question that arises is how to chunk the data. The criteria can be any of the following:<\/p>\n