A few months ago my friend Richard Swinbank posted a blog, More Get Metadata in ADF, about the limitations of using the Get Metadata activity in ADF to get information about files in a data lake. This to a twitter conversation as a bunch of other data engineers had been Read more…
I’ve worked with quite a few people new to Azure Data Factory, and one thing that seems to confuse new users is the difference between the developer sandbox where we build pipelines, and the published/deployed code. Understanding this is key to working with Git and using CI/CD pipelines to deploy Read more…
One of my clients has been adding data from multiple Salesforce instances to their data platform this week. One of their developers asked me if the Salesforce linked service could be made dynamic, as there is no place in the GUI to add parameters, or a dynamic values for the URL, user name or credentials. Here’s how we can do that.
When using the lookup activity in Azure Data Factory V2 (ADFv2), we have the option to retrieve either a multiple rows into an array, or just the first row of the result set by ticking a box in the UI. This allows us to either use the lookup as a Read more…