Return information about the current Magpie session.
Return information about the current Magpie session.
Magpie info
Execute a single command and return the result.
Execute a single command and return the result.
Execute a single command and return the result.
Commands return one of:
mc.execute("save result as table people")
command string to execute
command result
Get the data frame for a Magpie table, optionally qualified by schema.
Get the data frame for a Magpie table, optionally qualified by schema.
val df = mc.getTableDataFrame("people") df.show()
table name
table data frame
Returns the tokens that are potentially substitutable in the provided command.
Returns the tokens that are potentially substitutable in the provided command.
a string containing 0 or more substitution tokens in the format "$#{type:name}". The legacy/shorthand format for variables is also supported ("$#varname")
A de-duplicated list of tokens that are present 1 or more times in the provided command in the format "type:name". Variable tokens in the command that use the shorthand form are expanded to "variable:varname". Tokens are always lowercase
Checks if the provided input contains any tokens
Checks if the provided input contains any tokens
The string to test
True if the input string contains 1 or more replacable tokens
Execute the provided command and render the result visually
Execute the provided command and render the result visually
For example, to render the first 100 rows of the table people
:
mc.interpret("show 100 from people")
command to execute
Profile a data frame and render the profile visually
Profile a data frame and render the profile visually
mc.profile(df)
data frame
Replaces any substitute tokens in the given string with the token's name
Replaces any substitute tokens in the given string with the token's name
a string containing 0 or more tokens
a String with the tokens replaced by the name of the substitute reference by the token
Get the last result as a spark data frame
Get the last result as a spark data frame
mc.sql("select * from people") val people = mc.result.collect()
result data frame
SecretContext with methods for interacting with Magpie Secrets
Execute a SQL command, returning the resulting data frame.
Execute a SQL command, returning the resulting data frame.
val df = mc.sql("select distinct name from people")
SQL statement
data frame result
Inserts the provided substitutes into the provided command string.
Inserts the provided substitutes into the provided command string. Script comments will be stripped from the returned string.
Will throw a MagpieParsingException if no suitable substitute exists for a token
A string containing 0 or more substitute tokens, which may be in full or variable shorthand notation (See SubstitutionUtils.getTokens() for more details on token syntax)
A sequence of unencrypted substitute objects
A String with the substitute tokens replaced by their corresponding values
Inserts the provided substitutes into the provided text string
Inserts the provided substitutes into the provided text string
Will throw a MagpieParsingException if no suitable substitute exists for a token
A string containing 0 or more substitute tokens, which may be in full or variable shorthand notation (See SubstitutionUtils.getTokens() for more details on token syntax)
A sequence of unencrypted substitute objects
A String with the substitute tokens replaced by their corresponding values
Converts CapitalCamelCase to camelCase
Converts CapitalCamelCase to camelCase
to string to be converted
a camelCase string
VariableContext with methods for interacting with Magpie Variables
Unset a particular variable in the context.
Unset a particular variable in the context.
variable name
(Since version v1.21.0) Moved to mc.variables.remove()
Clear all variables in the context.
Clear all variables in the context.
(Since version v1.21.0) Moved to mc.variables.clear()
Get the value of a variable.
Get the value of a variable.
variable name
variable value
(Since version v1.21.0) Moved to mc.variables.get()
Determine whether a variable is defined on the context.
Determine whether a variable is defined on the context.
variable name
whether the variable is defined
(Since version v1.21.0) Moved to mc.variables.exists()
Get a listing of all variables defined on the context and their values.
Get a listing of all variables defined on the context and their values.
variable listing
(Since version v1.21.0) Use mc.variables.list()
instead
Set a variable to the given value in the context.
Set a variable to the given value in the context.
variable name
variable value
(Since version v1.21.0) Moved to mc.variables.set()
Substitute any variables present in the input string with current values stored in the context.
Substitute any variables present in the input string with current values stored in the context.
An exception is thrown if any variables are not able to be matched.
Note: substitution is performed automatically on arguments passed to MagpieContext.execute, MagpieContext.exec, and MagpieContext.sql.
input string
string with variables substituted for their values
(Since version v1.20.0) Variables are automatically substituted by the .exec, .execute, and .sql methods
Main entry point for Magpie functionality. The MagpieContext is used to execute commands, run SQL queries, get data frames for Magpie tables, and more.
Available as
mc
in scala script tasks and scala notebook blocks.Example usage: