Biml - Check Database Integrity Task

The SSIS Check Database Integrity Task allows you to verify the consistency of your databases. In this example, I'll check two specific database: AdventureWorksDW2012 and AdventureWorks2012. Via the DatabaseSelectionMode, you can specify whether it should check All, System, User or Specific databases.

Biml

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<AdoNetConnectionName="CM_ADO_DB"ConnectionString="Data Source=localhost\dev2012;Integrated Security=SSPI;Connect Timeout=30;Database=msdb;"Provider="SQL"/>
</Connections>

<Packages>
<PackageConstraintMode="Linear"Name="Task_CheckDatabaseIntegrity">
<Tasks>
<CheckDatabaseIntegrityConnectionName="CM_ADO_DB"DatabaseSelectionMode="Specific"Name="CHK Specific DBs"IgnoreDatabasesInNotOnlineState="true">
<Databases>
<Database>AdventureWorks2012</Database>
<Database>AdventureWorksDW2012</Database>
</Databases>
</CheckDatabaseIntegrity>
</Tasks>
</Package>
</Packages>
</Biml>

Results

Right clicking on that Biml file will result in a new package being built called Task_CheckDatabaseIntegrity.dtsx

The Control Flow will have a single Task in it along with our ADO.NET Connection Manager.

The Task itself allows you to select a connection manager and the database selection mode.

The details screen allows for the selection of specific databases as well as behaviour for offline mode.

Biml - Execute Package Task (Package Deployment Model)

The Execute Package Task allows you to run a package from within another package. Insert Xzibit meme here.

This post will show how to execute a package that uses the classic package deployment model but is still deployed in the msdb. Specifically, we'll create a Variable to hold the package name and path, an OLE DB Connection Manager and a package with an Execute Package Task. The Execute package Task will use an Expression to specify the package for execution. I'm going to use one of the native data collector packages to ensure the package passes validation.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<AdoNetConnectionName="CM_ADO_DB"ConnectionString="Data Source=localhost\dev2012;Integrated Security=SSPI;Connect Timeout=30;Database=msdb;"Provider="SQL"/>
</Connections>

<Packages>
<PackageConstraintMode="Linear"Name="Task_ExecutePackage_Package">
<Variables>
<VariableName="TargetPackage"DataType="String">\Data Collector\PerfCountersCollect</Variable>
</Variables>
<Tasks>
<ExecutePackageName="EPT Run Package"DelayValidation="true">
<SqlServerConnectionName="CM_OLE"PackagePath="Expression"></SqlServer>
<Expressions>
<ExpressionPropertyName="PackageName">@[User::TargetPackage]</Expression>
</Expressions>
</ExecutePackage>
</Tasks>
</Package>
</Packages>
</Biml>

Result

Right click on the biml file and you should generate a package named Task_ExecutePackage_Package.dtsx.
As you can see, the Control Flow has an Execute Package task as promised.

Double click on the execute package task, click the Package tab and it should be configured as shown.

The package itself is driven by the Variable @[User::TargetPackage] which you can see if you click on the Expressions tab, expand the chevron on Expressions under Misc and you will see the PackageName property is powered by our Variable.

Biml - Execute Package Task (Project Deployment Model)

The Execute Package Task allows you to run a package from within another package.

This post will show how to execute a package that uses the new project deployment model. This biml generates two packages. The child package, Task_ExecutePackage_ChildPackage, will have a Variable which has an Expression based on a Package Parameter. The second package, Task_ExecutePackage_Project, will have an Execute Package Task that is driven by Variable for Package Name. It will also showcase how to use Parameter Bindings to pass values to child packages.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<!--
    Create a pair of packages using the 2012+ project deployment model.
    Task_ExecutePackage_ChildPackage will have a package parameter mapped used as an expression
    Task_ExecutePackage_Project will reference the 
    -->
<Packages>
<PackageConstraintMode="Linear"Name="Task_ExecutePackage_ChildPackage">
<Parameters>
<ParameterDataType="String"Name="ParamValue">DesignTime</Parameter>
</Parameters>
<Variables>
<VariableDataType="String"Name="VariableValue"EvaluateAsExpression="true">@[$Package::ParamValue]</Variable>
</Variables>
</Package>
<PackageConstraintMode="Linear"Name="Task_ExecutePackage_Project">
<Variables>
<VariableName="TargetPackage"DataType="String">Task_ExecutePackage_ChildPackage</Variable>
<VariableName="ParameterValue"DataType="String">ParentValue</Variable>
</Variables>
<Tasks>
<ExecutePackageName="EPT Run Project Package"DelayValidation="true">
<ExternalProjectPackagePackage="Task_ExecutePackage_ChildPackage"></ExternalProjectPackage>
<ParameterBindings>
<ParameterBindingVariableName="User.ParameterValue"Name="ParamValue"></ParameterBinding>
</ParameterBindings>
<Expressions>
<ExpressionPropertyName="PackageName">@[User::TargetPackage]</Expression>
</Expressions>
</ExecutePackage>
</Tasks>
</Package>
</Packages>
</Biml>

Result

Right click on the biml file and you should generate packages named Task_ExecutePackage_ChildPackage.dtsx and Task_ExecutePackage_Project.dtsx.

Task_ExecutePackage_ChildPackage

The child package does nothing except have a Variable named "VariableValue" which has an Expression based on the Package Parameter of "@[$Package::ParamValue]".

Task_ExecutePackage_Project

Double click on the execute package task, click the Package tab and it should be configured as shown.

The Parameter Bindings tab shows the mapping between our local variable, ParameterValue, and the package's parameter name of ParamValue.

The package itself is driven by the Variable @[User::TargetPackage] which you can see if you click on the Expressions tab, expand the chevron on Expressions under Misc and you will see the PackageName property is powered by our Variable.

Biml - Execute Package Task File System (Package Deployment Model)

While I don't think there's a real difference between executing a package stored in SQL Server versus one stored on the file system, since the question came up on the Biml Forums.

This post contains the Biml to generate an Execute Package Task that runs a package on the file system. An Execute Package Task uses a File Connection Manager to point to the target package. Knowing that, you can make the execution configurable by either putting an Expression on the Connection Manager itself or setting the Expression within the Task itself. The following Biml demonstrates both approaches although you'd clearly only want to use one approach. I would use the Connection Manager approach if I was running packages in serial while I'd employ the Task level expression if I was running things in parallel.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<FileConnectionName="CM_File"FilePath="C:\Dropbox\Sandbox\BimlReference\BimlReference\Task_ExecutePackage_ChildPackage.dtsx"FileUsageType="ExistingFile"HasMultipleFiles="false"/>
</Connections>
<Packages>
<PackageConstraintMode="Linear"Name="Task_ExecutePackage_ChildPackage_FS">
</Package>

<PackageConstraintMode="Linear"Name="Task_ExecutePackage_Package_FS">
<Variables>
<VariableName="CurrentFileName"DataType="String">C:\Dropbox\Sandbox\BimlReference\BimlReference\Task_ExecutePackage_ChildPackage_FS.dtsx</Variable>
</Variables>
<Connections>
<ConnectionConnectionName="CM_File">
<Expressions>
<ExpressionPropertyName="CM_File.ConnectionString">@[User::CurrentFileName]</Expression>
</Expressions>
</Connection>
</Connections>
<Tasks>
<ExecutePackageName="EPT Run Package"DelayValidation="true">
<FileConnectionName="CM_File"></File>
<Expressions>
<ExpressionPropertyName="PackageName">@[User::CurrentFileName]</Expression>
</Expressions>
</ExecutePackage>
</Tasks>
</Package>
</Packages>
</Biml>

Result

Task_ExecutePackage_ChildPackage_FS

This is simply a package to ensure there's a valid package reference.

Task_ExecutePackage_Package_FS

Control flow

Execute Package Task

Package tab

The Package tab looks as expected, we have a file connection manager

File connection manager

I have an Expression on the File Connection Manager. It sets the ConnectionString property to our Variable @[User::CurrentFileName].

Biml - Execute SQL Server Agent Job

The Execute SQL Server Agent Job task allows you to run an agent job. It takes an ADO.NET connection and the name of the job to be executed. Unlike the backup database or check db task, this only allows for the specification of one job to executed.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<PackageConstraintMode="Linear"Name="Task_ExecuteSQLServerAgentJob">
<Tasks>
<ExecuteSqlAgentJob
AgentJob="syspolicy_purge_history"
ConnectionName="CM_ADO_DB"
Name="ESAJ Run Jobs">
</ExecuteSqlAgentJob>
</Tasks>
</Package>
</Packages>
</Biml>

Result

A package is created with an ADO.NET connection manager and an Execute SQL Server Agent Job task which fires the syspolicy_purge_history job. This should exist on servers so the package should pass validation.

Execute SQL Server Agent Job

The task is a simple one. Don't let the check box fool you, it's a radio button in that you can only pick one job.

Biml - Execute SQL Task

The Execute SQL Task allows an SSIS package to run an arbitrary SQL statement. You can parameterize it and return result sets as you please. This generates a package with three different connection type managers and Execute SQL Tasks using each of them. The Tasks demonstrate parameterization using the different syntax and how to pull back a result set. I'm going to create an SSIS Variable to hold the parameterized query, our parameter and our result set object.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<AdoNetConnectionName="CM_ADO_DB"ConnectionString="Data Source=localhost\dev2012;Integrated Security=SSPI;Connect Timeout=30;Database=msdb;"Provider="SQL"/>
<OdbcConnectionName="CM_ODBC"ConnectionString="Driver={SQL Server};Server=localhost\dev2012;Database=tempdb;Trusted_Connection=Yes;"/>
<OleDbConnectionName="CM_OLE"ConnectionString="Data Source=localhost\dev2012;Initial Catalog=tempdb;Provider=SQLNCLI11.0;Integrated Security=SSPI;"/>
</Connections>

<Packages>
<PackageConstraintMode="Linear"Name="Task_ExecuteSQL">
<Variables>
<VariableName="QueryADONet"DataType="String">SELECT D.Col1 FROM (SELECT @Parameter ) D(Col1) CROSS APPLY(VALUES (1), (2), (3)) C(C1);</Variable>
<VariableName="QueryODBC"DataType="String">SELECT D.Col1 FROM (SELECT ? ) D(Col1) CROSS APPLY(VALUES (1), (2), (3)) C(C1);</Variable>
<VariableName="QueryOLEDB"DataType="String">SELECT D.Col1 FROM (SELECT ? ) D(Col1) CROSS APPLY(VALUES (1), (2), (3)) C(C1);</Variable>
<VariableName="parameter"DataType="String">Hello world</Variable>
<VariableName="rsObject"DataType="Object"></Variable>
</Variables>

<Tasks>
<ExecuteSQL
ConnectionName="CM_ADO_DB"
Name="SQL ADO"
ResultSet="Full">
<VariableInputVariableName="User.QueryADONet"/>
<Parameters>
<Parameter
DataType="String"
VariableName="User.parameter"
Name="@Parameter"></Parameter>
</Parameters>
<Results>
<Result
Name="0"
VariableName="User.rsObject"/>
</Results>
</ExecuteSQL>

<!--
                This is not behaving as expected. 
                Not specifying a length results in 
"Invalid Precision Value Err w/ Character Field Parameters"
                http://support.microsoft.com/kb/132960
                The mapping between a string doesn't go smooth so
                we need to prvide the target data type explicitly.
                I did this by manually setting the data type in
                an existing SSIS package and viewing the code
                find SQLTask:DataType
                See also
                https://twitter.com/BimlScript/status/431798702403903489

                After fixing that, the next problem is the resultset.
                The query generates data correctly but the assignment
                back does not behave as expected.

                declare @p1 int
                set @p1=-1
                exec sp_prepexec @p1 output,N'@P1 varchar(50)',N'SELECT D.Col1 FROM (SELECT @P1 ) D(Col1) CROSS APPLY(VALUES (1), (2), (3)) C(C1);','H'
                select @p1                

-->
<ExecuteSQL
ConnectionName="CM_ODBC"
Name="SQL ODBC"
ResultSet="Full">
<VariableInputVariableName="User.QueryODBC"/>
<Parameters>
<Parameter
DataType="String"
VariableName="User.parameter"
DataTypeCodeOverride="12"
Length="50"
Name="1"></Parameter>
</Parameters>
<Results>
<Result
Name="0"
VariableName="User.rsObject"/>
</Results>
</ExecuteSQL>

<ExecuteSQL
ConnectionName="CM_OLE"
Name="SQL OLEDB"
ResultSet="Full">
<VariableInputVariableName="User.QueryOLEDB"/>
<Parameters>
<Parameter
DataType="String"
VariableName="User.parameter"
Name="0"></Parameter>
</Parameters>
<Results>
<Result
Name="0"
VariableName="User.rsObject"/>
</Results>
</ExecuteSQL>
</Tasks>
</Package>
</Packages>
</Biml>

Result

SQL ADO

General

Parameter Mapping

Result Set

SQL ODBC

General

Parameter Mapping

Result Set

SQL OLEDB

General

Parameter Mapping

Result Set

References

• Execute SQL Task
• Parameter mapping

Execute T-SQL Statement Task

The Execute T-SQL Statement Task is another of the Maintenance Tasks. As such, you'll need an ADO.NET connection to make this work.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<PackageConstraintMode="Linear"Name="ExecuteTSQLStatement">
<Tasks>
<ExecuteTSqlStatement
ConnectionName="CM_ADO_DB"
Name="ETS POC">
<SqlStatementSource>SELECT 100 As Foo;</SqlStatementSource>
</ExecuteTSqlStatement>
</Tasks>
</Package>
</Packages>
</Biml>

Result

Execute T-SQL Statement Task

Biml - Expression Task

The Expression Task, introduced with the 2012 release of SSIS, allows for the assignment of a value to an SSIS Variable. Prior to the 2012 release, you either needed to use a trivial, but custom component or perform this in a Script Task.

You'll find this Expression Task is a bit of a pain to work with. A classic expression for building a string of "YYYY-MM-DD" is (DT_WSTR, 4) YEAR(@[System::StartTime]) + "-" + RIGHT("0" + (DT_WSTR, 2) MONTH(@[System::StartTime]), 2) + "-" + RIGHT("0" + (DT_WSTR, 2) DAY(@[System::StartTime]), 2) Since that expression has double quotes in it and we'll need to be storing that string inside a string, we need to encode the expression's double quotes with their entity name "

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<PackageConstraintMode="Linear"Name="Task_Expression">
<Variables>
<VariableName="DateString"DataType="String"></Variable>
<VariableName="Math"DataType="Int32">0</Variable>
<VariableName="Reference"DataType="String"EvaluateAsExpression="true">(DT_WSTR, 4)  YEAR(@[System::StartTime]) + "-" + RIGHT("0" + (DT_WSTR, 2)  MONTH(@[System::StartTime]), 2) + "-" + RIGHT("0" + (DT_WSTR, 2) DAY(@[System::StartTime]), 2)</Variable>
</Variables>
<Tasks>
<Expression
Name="EXP Add"
Expression="@[User::Math]=1+4"
>
</Expression>
<Expression
Name="EXP Assign file name"
Expression="@[User::DateString] = (DT_WSTR, 4)  YEAR(@[System::StartTime]) + &quot;-&quot; + RIGHT(&quot;0&quot; + (DT_WSTR, 2)  MONTH(@[System::StartTime]), 2) + &quot;-&quot; + RIGHT(&quot;0&quot; + (DT_WSTR, 2) DAY(@[System::StartTime]), 2)">
</Expression>
</Tasks>
</Package>
</Packages>
</Biml>

Result

EXP Add

Here we're performing addition and assigning the result to @[User::Math]

EXP Assign file name

In this task, we're computing a date string. You'll see the double quote entity is correctly represented.

Biml - Execute Process Task

The Execute Process Task allows you to invoke any executable. While it should be obvious, don't run programs that have graphical user elements.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<PackageConstraintMode="Linear"Name="Task_ExecuteProcess">
<Variables>
<VariableDataType="String"Name="StandardIn"></Variable>
<VariableDataType="String"Name="StandardOut"></Variable>
<VariableDataType="String"Name="StandardErr"></Variable>
<VariableDataType="String"Name="Executable">J:\Windows\System32\ipconfig.exe</Variable>
</Variables>
<Tasks>
<ExecuteProcess
Executable="J:\Windows\System32\ipconfig.exe"
Name="EP Ipconfig"
WorkingDirectory="J:\windows\System32"
StandardInputVariableName="User.StandardIn"
StandardErrorVariableName="User.StandardErr"
StandardOutputVariableName="User.StandardOut"
>
<Expressions>
<ExpressionPropertyName="Executable">@[User::Executable]</Expression>
</Expressions>

</ExecuteProcess>
</Tasks>
</Package>
</Packages>
</Biml>

Result

A package is generated that calls ipconfig and assigns the results to standard out and standard error SSIS Variables.

EP Ipconfig

Process

As usual, I have everything driven by SSIS Variables. The "Executable" appears to be hard coded but you'll see there's actually an expression on it.

Expressions

Execution Results

You can see the results of executing the generated package. The output of ipconfig is assigned to our Variable User::StandardOut

Biml - File System Task

The File System Task allows you to manipulate objects within the file system without resorting to a Script Task or the Execute Process Task.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<PackageConstraintMode="Linear"Name="Task_FileSystem">
<Variables>
<VariableName="FileName"DataType="String">system.ini</Variable>
<VariableName="FolderBase"DataType="String">J:\SSISData</Variable>
<VariableName="FocusArea"DataType="String">Sales</Variable>
<VariableName="FolderInput"DataType="String"EvaluateAsExpression="true">@[User::FolderBase] + "\\" + @[User::FocusArea] + "\\Input"</Variable>
<VariableName="FolderArchive"DataType="String"EvaluateAsExpression="true">@[User::FolderBase] + "\\" + @[User::FocusArea] + "\\Archive"</Variable>
<VariableName="FolderOutput"DataType="String"EvaluateAsExpression="true">@[User::FolderBase] + "\\" + @[User::FocusArea] + "\\Output"</Variable>
<VariableName="FileSource"DataType="String"EvaluateAsExpression="true">"J:\\Windows" + "\\" + @[User::FileName]</Variable>
<VariableName="FileDestination"DataType="String"EvaluateAsExpression="true">@[User::FolderArchive] + "\\" + @[User::FileName]</Variable>
<VariableName="FileRenamed"DataType="String"EvaluateAsExpression="true">@[User::FolderArchive] + "\\MaximumOutput.txt"</Variable>
<VariableName="OverwriteDestinationFile"DataType="Boolean">True</Variable>
</Variables>

<Tasks>
<FileSystem
Operation="CreateDirectory"
Name="FS CreateDirectory">
<VariableInputVariableName="User.FolderArchive"/>
<Annotations>
<AnnotationAnnotationType="Description">Create a folder based on our Variable</Annotation>
</Annotations>
<Expressions>
<ExpressionPropertyName="OverwriteDestinationFile">@[User::OverwriteDestinationFile]</Expression>
</Expressions>
</FileSystem>

<FileSystem
Operation="CopyFile"
Name="FS CopyFile">
<VariableInputVariableName="User.FileSource"/>
<VariableOutputVariableName="User.FolderArchive"/>
<Annotations>
<AnnotationAnnotationType="Description">Copy a file from source to destination</Annotation>
</Annotations>
<Expressions>
<ExpressionPropertyName="OverwriteDestinationFile">@[User::OverwriteDestinationFile]</Expression>
</Expressions>
</FileSystem>

<FileSystem
Operation="RenameFile"
Name="FS RenameFile">
<VariableInputVariableName="User.FileDestination"/>
<VariableOutputVariableName="User.FileRenamed"/>
<Annotations>
<AnnotationAnnotationType="Description">Rename the copied file to something else</Annotation>
</Annotations>
<Expressions>
<ExpressionPropertyName="OverwriteDestinationFile">@[User::OverwriteDestinationFile]</Expression>
</Expressions>
</FileSystem>

<FileSystem
Operation="DeleteDirectoryContent"
DelayValidation="true"
Name="FS DeleteDirectoryContent">
<VariableInputVariableName="User.FolderArchive"/>
<Annotations>
<AnnotationAnnotationType="Description">Delete everything in the folder</Annotation>
</Annotations>
</FileSystem>

<FileSystem
Operation="DeleteDirectory"
Name="FS DeleteDirectory">
<VariableInputVariableName="User.FolderArchive"/>
<Annotations>
<AnnotationAnnotationType="Description">Remove the folder we just created</Annotation>
</Annotations>
</FileSystem>

</Tasks>
</Package>
</Packages>
</Biml>

Result

That mess of biml creates 5 file system tasks. We create a set of folders \SSISData\Sales\Archive. Into the Archive folder, we will copy the system.ini file from the Windows folder. The next task renames it to MaximumOutput only to have all the contents of the Archive folder deleted prior to completely deleting the folder.

FS CreateDirectory

Create a directory, if it does not exist based on our Variable @[User::FolderArchive]

FS CopyFile

Using variables, we will copy our file, defined as @[User::FileSource] to our @[User::FolderArchive]

FS RenameFile

Rename our file from @[User::FileDestination] to @[User::FileRenamed]

FS DeleteDirectoryContent

Purge all the files in @[User::FolderArchive]

FS DeleteDirectory

Finally we drop the folder @[User::FolderArchive]

Biml - FTP Task

SSIS has some really awesome tasks to improve your ability to acquire, manage and push data about. The FTP Task isn't one of them.

I cut my teeth with FTP on a command line so the limited options available via the native task are rather bothersome. Compound that with some of the other annoyances, like erroring when a file does not exist, you can see why I prefer to use a command file.

In this example, we are going to access a publicly available ftp server using anonymous user and blank password. We'll download a file into our working folder and then delete it.

As you might have observed throughout my postings, I really believe the power of SSIS comes through using Expressions. Something that surprised me was that as of this writing, I cannot use an Expression on the OperationName property of an FTP task. Well, I can apply one, but I have no idea what the text string that should go inside of one of those. Instead, by editing an existing package's operationg via Visual Studio and then viewing the resulting XML, I was able to discern the correlation between the displayed text and the underlying Operation numeric value.

0

Send file

1

Receive file

2

Delete Local Files

3

Delete Remote Files

4

Create Local Directory

5

Create Remote Directory

6

Remove Local Directory

7

Remove Remote Directory

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<FtpConnectionName="CM_FTP"ServerName="localhost"UserName="user"Password="pass"ServerPort="21"UsePassiveMode="true"/>
</Connections>
<Packages>
<PackageConstraintMode="Linear"Name="Task_FTP">
<Connections>
<ConnectionConnectionName="CM_FTP">
<Expressions>
<ExpressionPropertyName="CM_FTP.ServerName">@[User::ServerName]</Expression>
<ExpressionPropertyName="CM_FTP.ServerPort">@[User::ServerPort]</Expression>
<ExpressionPropertyName="CM_FTP.ServerUserName">@[User::ServerUserName]</Expression>
<!--<Expression PropertyName="CM_FTP.ServerPassword">@[User::ServerPassword]</Expression>-->
<ExpressionPropertyName="CM_FTP.Timeout">@[User::Timeout]</Expression>
<ExpressionPropertyName="CM_FTP.UsePassiveMode">@[User::UsePassiveMode]</Expression>
</Expressions>
</Connection>
</Connections>
<Variables>
<VariableName="ServerName"DataType="String">ftp2.census.gov</Variable>
<VariableName="ServerPort"DataType="Int32">21</Variable>
<VariableName="ServerUserName"DataType="String">anonymous</Variable>
<VariableName="ServerPassword"DataType="String">
<Annotations>
<AnnotationAnnotationType="Description">Don't ever do this</Annotation>
</Annotations>
</Variable>
<VariableName="Timeout"DataType="Int32">60</Variable>
<VariableName="UsePassiveMode"DataType="Boolean">False</Variable>
<VariableName="LocalFolder"DataType="String">C:\ssisdata</Variable>
<VariableName="LocalFile"DataType="String">p3p.xml</Variable>
<VariableName="RemoteFolder"DataType="String">/w3c</Variable>
<VariableName="RemoteFile"DataType="String">p3p.xml</Variable>
<!-- 
                0 Send file 
                1 Receive file
                2 Delete Local Files
                3 Delete Remote Files
                4 Create Local Directory
                5 Create Remote Directory
                6 Remove Local Directory
                7 Remove Remote Directory
                -->
<VariableName="Operation"DataType="Int32">1</Variable>
<!--
                The text I have been using does not seem to align with values
                -->
<VariableName="OperationName"DataType="String">Receive</Variable>
<VariableName="OperationName2"DataType="String">Receive files</Variable>
<VariableName="RemotePath"DataType="String"EvaluateAsExpression="true">@[User::RemoteFolder] + "/" + @[User::RemoteFile]</Variable>
<VariableName="LocalPath"DataType="String"EvaluateAsExpression="true">@[User::LocalFolder] + "\\" + @[User::LocalFile]</Variable>
<VariableName="IsTransferTypeASCII"DataType="Boolean">True</Variable>
<VariableName="OverwriteDestination"DataType="Boolean">True</Variable>
</Variables>
<Tasks>
<Ftp
ConnectionName="CM_FTP"
Operation="Send"
Name="FTP Receive File"
IsTransferTypeAscii="false"
OverwriteDestination="true"
>
<VariableInputVariableName="User.LocalFolder"></VariableInput>
<VariableRemotePathVariableName="User.RemotePath"/>
<Expressions>
<!-- I must not be using the right text here-->
<!--<Expression PropertyName="OperationName">@[User::OperationName]</Expression>-->
<ExpressionPropertyName="Operation">@[User::Operation]</Expression>
<ExpressionPropertyName="IsTransferTypeASCII">@[User::IsTransferTypeASCII]</Expression>
<ExpressionPropertyName="OverwriteDestination">@[User::OverwriteDestination]</Expression>
</Expressions>
</Ftp>

<Ftp
ConnectionName="CM_FTP"
Operation="Send"
Name="FTP Delete downloaded file">
<VariableInputVariableName="User.LocalPath"></VariableInput>
<Expressions>
<ExpressionPropertyName="Operation">2</Expression>
</Expressions>
</Ftp>

<Ftp
ConnectionName="CM_FTP"
Operation="Send"
Name="FTP Receive Expression on OperationName"
IsTransferTypeAscii="false"
OverwriteDestination="true"
>
<VariableInputVariableName="User.LocalFolder"></VariableInput>
<VariableRemotePathVariableName="User.RemotePath"/>
<Expressions>
<!-- I must not be using the right text here-->
<ExpressionPropertyName="OperationName">@[User::OperationName]</Expression>
<ExpressionPropertyName="IsTransferTypeASCII">@[User::IsTransferTypeASCII]</Expression>
<ExpressionPropertyName="OverwriteDestination">@[User::OverwriteDestination]</Expression>
</Expressions>
</Ftp>
</Tasks>
</Package>
</Packages>
</Biml>

Result

A package is created with an FTP connection manager, expressions applied to it, and 3 FTP Tasks.

FTP Receive File

A task is created to receive, GET, a file. This operation type is set directly in the Biml as a Send but we later override it via Expresions.

Expressions

We're going to set the operation type by the numeric value of the Operation property.

FTP Delete downloaded file

An FTP task set to Delete Local Files

FTP Receive Expression on OperationName

This step is identical to the first except in our application of Expressions.

Expressions

You'll observe the Operation remains "Send files" despite setting OperationName to either the value of "Receive" or "Receive files"

Biml - History Cleanup Task

The History Cleanup Task is another of those Database Maintenance tasks that I never use. It requires an ADO connection manager and allows you to cleanup any combination of backup and restore history, SQL Server Agent history and Maintenance plan history.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<AdoNetConnectionName="CM_ADO_DB"ConnectionString="Data Source=localhost\dev2012;Integrated Security=SSPI;Connect Timeout=30;Database=msdb;"Provider="SQL"/>
</Connections>

<Packages>
<PackageConstraintMode="Linear"Name="Task_HistoryCleanup">
<Tasks>
<!-- Look busy without doing anything -->
<HistoryCleanup
ConnectionName="CM_ADO_DB"
Name="HC Clean as my children do"
OlderThanTimeUnits="4"
OlderThanTimeUnitType="Weeks"
RemoveBackupRestoreHistory="false"
RemoveDBMaintenanceHistory="false"
RemoveSqlAgentHistory="false"
/>
</Tasks>
</Package>
</Packages>
</Biml>

Result

A package is created with our ADO.NET connection manager and a single Cleanup History task.

HC Clean as my children doa

I've configured the History Cleanup task to do nothing.

I love that it actually warns you of this as you click the View T-SQL button

Biml - Maintenance Cleanup Task

The Maintenance Cleanup Task removes artifacts from maintenance plans, backups, reports, etc. As with other Maintenace Tasks, this requires an ADO.NET Connection Manager.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<AdoNetConnectionName="CM_ADO_DB"ConnectionString="Data Source=localhost\dev2012;Integrated Security=SSPI;Connect Timeout=30;Database=msdb;"Provider="SQL"/>
</Connections>

<Packages>
<PackageConstraintMode="Linear"Name="Task_MaintenanceCleanup">
<!-- 
                Backup\InstanceName\DBName\DIFF 
                Backup\InstanceName\DBName\FULL 
                Backup\InstanceName\DBName\LOG 
            -->
<Variables>
<VariableName="FileExtension"DataType="String">bacon</Variable>
<VariableName="FolderBase"DataType="String">J:\Backup</Variable>
<VariableName="InstanceName"DataType="String">WESTMARCH$DEV2012</Variable>
<VariableName="DatabaseName"DataType="String">Adventureworks2012</Variable>
<VariableName="FolderPath"DataType="String"EvaluateAsExpression="true">@[User::FolderBase] + "\\" + @[User::InstanceName] + "\\" + @[User::DatabaseName] </Variable>
</Variables>
<Tasks>
<!--
                    http://stackoverflow.com/questions/11571002/can-someone-help-explain-some-of-the-new-ssis-2012-properties 
                -->
<MaintenanceCleanup
ConnectionName="CM_ADO_DB"
Name="MC Clean backup files"
DeleteFromAllBricks="true"
AgeBased="true"
FileTypeSelected="BackupFiles"
OlderThanTimeUnits="4"
OlderThanTimeUnitType="Weeks"
>
<!-- Delete all the bacon -->
<FolderAndExtension
Folder="C:\DrivenByVariable"
FileExtension="VariableToo"
IncludeFirstLevelSubfolders="true"
>
</FolderAndExtension>
<Expressions>
<ExpressionPropertyName="FolderPath">@[User::FolderPath]</Expression>
<ExpressionPropertyName="FileExtension">@[User::FileExtension]</Expression>
</Expressions>
</MaintenanceCleanup>
</Tasks>
</Package>
</Packages>
</Biml>

Result

A package is created with Variables to control the FileExtension and FolderPath that are removed. I'm basing the three Variables that build my FolderPath variable based on the default output of Ola's Maintenance plans.

MC Clean Backup Files

Great googly-moogly, someone's attempting to get rid of all the bacon! Well, at least any Backup files that are in the first-level subfolder, with an extension of bacon, that are older than 4 weeks.

Expressions

As you can see, I've applied an Expression to control both FileExtension and FolderPath based on our user Variables.

Biml - Message Queue Task

Why have one tool when you can multiples? In this case, queuing technologies. As a SQL Server professional, I was aware of Service Broker but there's a whole native queuing technology built into Windows: Microsoft Message Queuing. Until doing the research on the SSIS Message Queue Task, I had assumed it pulled from Service Broker. Yeah, well assumptions...

MSMQ Setup

I originally created a private queue on a different box than I was running my package on but the native Message Queue Task can't converse with a remote server. MSDN has a write up on how you can use a script task to talk to a remote private queue

Biml

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<MsmqConnectionName="CM_MSMQ"Path=".\private$\POC"/>
</Connections>

<Packages>
<PackageConstraintMode="Linear"Name="Task_MessageQueue">
<Variables>
<VariableName="MessageSource"DataType="String">What God hath wrought</Variable>
<VariableName="MessageReceipt"DataType="String"></Variable>
</Variables>
<Tasks>
<MessageQueue
MsmqConnectionName="CM_MSMQ"
Name="MQ Send Message"
>
<VariableInputVariableName="User.MessageSource"></VariableInput>
<Expressions>
<ExpressionPropertyName="MessageString">@[User::MessageSource]</Expression>
</Expressions>
</MessageQueue>

<MessageQueue
MsmqConnectionName="CM_MSMQ"
Name="MQ Receive Message"
RemoveFromQueue="true"
ReceiveMessageTimeOut="30"
>
<StringVariableOutputVariableName="User.MessageReceipt"/>
</MessageQueue>
</Tasks>
</Package>
</Packages>
</Biml>

Result

A package is created with a pair of Message Queue Tasks. One to send and one to receive. There are two Variables: MessageSource and MessageReceipt, both of type String. MessageSource is preloaded with a value of What God hath wrought. After execution, if all went well, that value will be loaded into MessageReceipt. All of this will use our MSMQ connection manager, CM_MSMQ.

CM_MSMQ

Here's a screenshot of what the connection manager looks like. The value of the path is .\private$\POC to indicate the message queue server is local, it's a private queue named POC.

MQ Send Message

General tab

We have set the task to send a message

Send tab

Sent tab is set to send an unencrypted message. While the StringMessage property appears to be hard coded, that's just an artifact of how the UI works.

Expressions

MQ Receive Message

A receive message task will pull messages off the queue and assign their value to our Variable.

General tab

Receive tab

In the Receive tab, we remove them from the source queue and assign the message into our Variable User::MessageReceipt.

Execution Results

Green is good. Even better is that the value of @[User::MessageReceipt] matches @[User::MessageSource]

Biml - Notify Operator Task

The maintenance task Notify Operator contacts the pre-defined operators. It requires an ADO.NET connection manager to work. Specify the operator, subject and message contents and away you go.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<AdoNetConnectionName="CM_ADO_DB"ConnectionString="Data Source=localhost\dev2012;Integrated Security=SSPI;Connect Timeout=30;Database=msdb;"Provider="SQL"/>
</Connections>

<Packages>
<PackageConstraintMode="Linear"Name="Task_NotifyOperator">
<Variables>
<VariableName="OperatorName"DataType="String">Ops</Variable>
<VariableName="Subject"DataType="String">Look here</Variable>
<VariableName="Message"DataType="String">Things have gone south</Variable>
<VariableName="Query"DataType="String"EvaluateAsExpression="true">"
DECLARE @OpName sysname = 'Ops';

IF NOT EXISTS
(
    SELECT * 
    FROM msdb.dbo.sysoperators AS SO 
    WHERE SO.name = @OpName
)
EXECUTE msdb.dbo.sp_add_operator 
    @name = @OpName
,   @enabled = 1
,   @pager_days = 0
,   @email_address = N'spam@billfellows.net';
"</Variable>
</Variables>
<Tasks>
<ExecuteSQL
ConnectionName="CM_ADO_DB"
Name="SQL Create Operator">
<VariableInputVariableName="User.Query"/>
</ExecuteSQL>

<NotifyOperator
ConnectionName="CM_ADO_DB"
Name="NO Ack">
<OperatorsToNotify>
<Operator>Ops</Operator>
</OperatorsToNotify>
<Subject>Ack</Subject>
<Message>Body of the notification</Message>
<Expressions>
<ExpressionPropertyName="Subject">@[User::Subject]</Expression>
<!--<Expression PropertyName="OperatorName">@[User::Subject]</Expression>-->
<ExpressionPropertyName="Message">@[User::Message]</Expression>
</Expressions>
</NotifyOperator>
</Tasks>
</Package>
</Packages>
</Biml>

Result

A package is created with an Execute SQL Task to ensure our Operator exists and then an actual Notify Operator Task is dropped into our Control Flow.

NO Ack

This is our Notify Operator task and it acknowledges the errors on the server.

Expression

I could figure out how to configure the message and the subject but couldn't figure out how to set the Operator name via expression.

Biml - RebuildIndex Task

The maintenance task, Rebuild Index Task, described with Biml. By now you know the drill, it requires an ADO.NET Connection Manager to work.

<Bimlxmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package
ConstraintMode="Linear"
Name="Task_RebuildIndex">
<Tasks>
<RebuildIndex
ConnectionName="CM_ADO_DB"
DatabaseSelectionMode="All"
ObjectSelectionMode="Tables"
Name="RI All Tables">
</RebuildIndex>

<!-- 
                    Rebuild the index on a specific view
                -->
<RebuildIndex
ConnectionName="CM_ADO_DB"
DatabaseSelectionMode="Specific"
ObjectSelectionMode="Views"
Name="RI Specific View"
ReindexPercentage="50"
ReindexWithOriginalAmount="false"
SortInTempDB="false"
SkipUnsupported="true"
KeepOnline="true">
<Databases>
<Database>AdventureWorks2012</Database>
</Databases>
<Objects>
<Object>Production.vProductAndDescription</Object>
</Objects>
</RebuildIndex>
</Tasks>
</Package>
</Packages>
</Biml>

Result

The above Biml describes a package that creates two Rebuild Index Tasks. The first rebuilds on all the tables while the second targets a specific indexed view.

RI All Tables

Here we reindex all the tables in all the databases.

RI Specific View

This snippet of Biml describes reindexing a specific view, Production.vProductAndDescription in the AdventureWorks2012 database.

Specify Database

Specify the database(s) we should use. Normally, I'd have trimmed out the empty space at the top of the window there but I left it as is to point out the wasted real estate. It's like there's a missing title block.

Specify Object

Here we're selecting the one object we're interested in re-indexing.

Problem definition

If: A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
Is equal to;
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

Then

H+A+R+D+W+O+R+K ;
8+1+18+4+23+15+18+11=98%

K+N+O+W+L+E+D+G+E ;
11+14+15+23+12+5+4+7+5=96%

L+O+V+E;
12+15+22+5 = 54%

L+U+C+K ;
12+21+3+11 =47%

None of them makes 100%.

Then what makes 100% ???

Is it Money?
NO !!! M+O+N+E+Y= 13+15+14+5+25=72%

Leadership?
NO !!! L+E+A+D+E+R+S+H+I+P= 12+5+1+4+5+18+19+8+9+16=97%

Every problem has a solution, only if we perhaps change our "ATTITUDE".

A+T+T+I+T+U+D+E ;
1+20+20+9+20+21+4+5 = 100%


It is therefore OUR ATTITUDE towards Life and Work that makes OUR Life 100% Successful.. 

What makes 100%?

Python to the rescue

Capital A is ASCII value 65; Z is 90; a is 97 and z is 122. Knowing this, if I make everything uppercase, convert each letter to its ordinal value and subtract 64, and then add up all the values, I should be able to identify the value of a particular word. Piece of cake.

# create a variable to hold our translation between ASCII and percent value
offset = 64

def WordValue(word):
    # compute the value for each character
    # convert to upper case and subtract 64 to get it to 1-26
    return sum([(ord(c)-offset) for c in word.upper()])

# Create a word list. I cribbed from http://www.manythings.org/vocabulary/lists/l/
f  = open('/Users/bfellows/qualities.txt')

# Create an empty list to hold all matching words. Since my source has duplicates,
# I will want to filter them out
l = []

# rip through my file line by line
for line in f.read().splitlines():
    # Test whether our summed value matches the target
    if WordValue(line) == 100:
        # keep track of everything that matches
        l.append(line)

# Enumerate through all of our matches
for item in set(l):
    # Display matches to the screen
    print item

Source data

a
an
able
about
above
abuse
accept
accident
accuse
across

What makes 100%?

• prevent
• telescope
• Congress
• hospital
• ornament
• telephone
• boycott
• culture
• inflation
• excellent
• writing
• interfere
• repress
• lightning

It seems that every quarter there's always some heartache and disappointment from those that didn't make the cut. I know I kept hoping I'd just magically show up on Microsoft's radar and *poof*, MVP. Now that I know more about how this works, I thought it might be helpful to share it with others.

What is the process?

Phase 2

Phase 3

Why me

What I thought qualified me from a quantitative perspective was

• blogging
• presenting
• organizing
• stackoverflowing

Infographic

Takeaways

• Speaking Engagements
• User Group Participation (include presentations under Speaking)
• Event Organizer or Chair (other than user group leader/organizer)
• Forum Activity
• Blogging
• Publishing
• Twitter/Social Media
• Other

For Speaking engagements, they will want to know how many people were there. For UG activity, they want to know total group size, frequency of meetings, and average attendance. Event Organizer - how many people attended. Forums, they are interested in quantity of answers plus any answers that you wanted to provide as a highlight. Blogging, they care about your numbers reached per month. Publishing, what and how many things? Twitter/social media, it only asks how many followers you had an links to your account.

That was a pain the backside to pull some of that data together. I had a general idea of how many people were at my UG/SQL Saturday presentations, but was that room big? It seemed full but was full 25 people or 45? While data.stackexchange.com was a cute way to get data out of the site, it still left me wanting as I had to change my user id per site.

Beyond that, since my recipe didn't work, I can't say what else is needed if you want to be an MVP. I am thankful for everyone who helped me along the way. And to those newly awarded or renewed, a hearty congratulations. For those that were not renewed, thank you for your efforts. #mvpbuzz

Bring out your dead!

Just a quick post to share the script. At my current client, they have a large codebase that's not been cared for. I give them credit for generally avoiding dynamic SQL but the proliferation of autogenerated code has resulted in thousands, not an exaggeration, of objects that were generated and never used. I had run down fixing a dependency chain only to learn the rootmost object was broken as the day is long.

There has to be a better way of figuring this out. SQL Server Data Tools you say? Yeah, it took 2 days to reverse engineer the database and validate the dependencies only to run out of memory. I settled on sys.sp_refreshsqlmodule to at least give me a fighting chance of finding out what's broken in the database. The following query generates a list of all non-schemabound objects and then calls sys.sp_refreshmodules against the objects. if it raises an error, I drop it into a table so I can inspect it later as well as print out an error message. Quick and dirty, there's probably some logic issues in there, things that aren't dead yet, but this at least helps me triage the database.

-- This script attempts to generate a report on whether all the SQL modules are valid
-- in a database. This can generate false positives as there is no intelligence
-- built into the order to ensure object dependencies are followed.
SET NOCOUNT ON;
DECLARE
    @SchemaName sysname
,   @ObjectName sysname
,   @QualifiedObjectName sysname
,   @QueryTemplate nvarchar(1000) = N'EXECUTE sys.sp_refreshsqlmodule @name;'
,   @ParamList nvarchar(20) = N'@Name sysname';


DECLARE ObjectCursor CURSOR READ_ONLY
FOR
SELECTALL
    SCHEMA_NAME(S.schema_id) AS SchemaName
,   S.name AS ObjectName
FROM
    sys.objects AS S
INNERJOIN
    (
-- schema bound objects can't get recompiled
SELECT
            SM.object_id
FROM
            sys.sql_modules AS SM
WHERE
            SM.is_schema_bound = 0
    ) AS SM
ON SM.object_id = S.object_id
WHERE
    S.is_ms_shipped = 0
-- adjust this list as needed
AND S.type in ('FN', 'IF', 'IT', 'P', 'TF', 'TR', 'V')
ORDERBY S.schema_id
;

DECLARE @ManualCleanup TABLE
(
    SchemaName sysname
,   ObjectName sysname
,   SQLStatement nvarchar(1000) NOTNULL
);

OPEN ObjectCursor

FETCHNEXT
FROM ObjectCursor 
INTO @SchemaName
,   @ObjectName;

WHILE (@@fetch_status > -1)
BEGIN
SET @QualifiedObjectName = QUOTENAME(@schemaName) + '.' + QUOTENAME(@ObjectName);
BEGIN TRY
EXECUTE dbo.sp_executesql @queryTemplate, @ParamList, @Name = @QualifiedObjectName;
END TRY
BEGIN CATCH
IF@@TRANCOUNT> 0 ROLLBACKTRANSACTION;
INSERTINTO
            @ManualCleanup
        (
            SchemaName
        ,   ObjectName
        ,   SQLStatement
        )
SELECT
            @SchemaName
        ,   @ObjectName
        ,   'EXECUTE sys.sp_refreshsqlmodule @name = N''' + @QualifiedObjectName + ''';';
END CATCH

FETCHNEXT
FROM ObjectCursor 
INTO @SchemaName
    ,   @ObjectName;
END

CLOSE ObjectCursor
DEALLOCATE ObjectCursor


SELECT
    MC.* 
FROM
    @ManualCleanup AS MC;

<Connection
Name="CM_OLE"
ConnectionString="Data Source=.\dev2014;Initial Catalog=tempdb;Provider=SQLNCLI11.1;Integrated Security=SSPI;"
/>

Resolution