AWS Lake Formation, a service to build, secure, and manage data lakes on AWS, is now generally available in US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), and Asia Pacific (Tokyo).
Lake Formation was introduced at last year’s AWS re:Invent conference as a way of ingesting and processing data, preparing it for analysis and machine learning. The product is built on the capabilities the extract, transform, and load service AWS Glue offers, which classifies data and stores central information such as schemes and metadata in a central catalog. It also assists in getting rid of data duplicates and converting data into a suitable format.
To get going, customers can use the Lake Formation console to define the data sources to be used for their data lake. This can be sped up by choosing one of the blueprints the service provides to automatically load data from the most common databases and log providers. Once this is set up, users can decide on data access and security policies that should be applied. Those will then make sure data is only available to those allowed to look at it and properly encrypted where needed.
The data lake can be accessed through some of AWS’ analytics and machine learning offerings for further processing. Insight into which jobs or operations are running at any given time is available through the Lake Formation console.
The announcement was put out on the same day as AWS introduced a preview of its new Tools for PowerShell. Since its first release, the number of cmdlets the toolkit spans has gone up to almost 6000 across around 160 services, which naturally meant a significant increase in the time needed to import the module.
In the refactored version, each AWS service has its own PowerShell module, so that users are free to install only the ones they actually need, reducing startup time and getting rid of the need to explicitly import preview modules before use. The team behind the toolkit took the chance to remove some obsolete cmdlets and add markings for mandatory parameters, which seems to have gotten a lot of requests. More details can be found in a blog post.