We always hesitate to ask Dyalog for an enhancement, for fear that while we think it is important at the time, in the long run it will turn out to be a mistake or not needed. We have users too, and we see this often from the supplier side. People often ask for things they really don't need.

When ⎕CSV was being designed, we insisted on the invert variant for character columns to be imported as matrices rather than vectors of vectors. We wanted this because nesting is expensive, and takes up a lot of space. And because our custom DBMS is optimized for fixed-width character columns. This turned out to be a mistake.

Dysfunctional CSV files are the rule, not the exception. It's easy to find examples of files with columns that are mostly empty, but have a few entries thousands of characters wide. If you run into a large file with such a column, it's game over (WSFULL) if you use the invert variant. Throw in a few characters of ⎕DR 160 and the problem is even worse.

If the CSV file is small, nesting is not an issue. If the CSV is large, it must be read in blocks and there is no way to know what the widest value will be until you have read the entire file.

Thus, it is better to always read a block of a column as a nested vector, inspect the length of each value, and only invert the column block if the matrix form is smaller than the nested form. In addition we can specify am optional column width limit to simply truncate any wide data:

ProcessColumn←{
     w←⌈/≢¨⍵
     m←⍺.ColumnWidthLimit
     (m>0)∧w≤m:↑⍵
     (m>0)∧w>m:↑m↑¨⍵
     (w×≢⍵)<⎕SIZE'⍵':↑⍵
     ⍵
 }

If, in addition, we store the size of the nested column block and the maximum width, the most efficient space representation of the column can be determined at the end.

We over-complicated the design of ⎕CSV for no good reason.

In Abacus we have a function for setting the value of a component:

SetComponentValue←{
     ⍝ ⍺ ←→ Component
     ⍝ ⍵ ←→ Value
     ⍺ ⍺.Class.SetValue ⍵
 }

The left argument is the component, so we need to have the component in hand before calling this function.

Abacus does not use Dyalog classes, but the "instance" ⍺ has a reference to the component namespace that contains the SetValue function.

The component is often retrieved via its name from the document or a node using GetComponent, which takes a node as the left argument and the name as the right argument, so this pattern is common:

    C←D GetComponent 'MyComponentName' 
    C SetComponentValue 'MyValue'

It was suggested that SetComponentValue should be able to take a component name as its argument to avoid calling GetComponent. Of course it would also need know about the document or node. A first pass produced:

SetComponentValue←{
     ⍝ ⍺ ←→ Component|Node and Name
     ⍝ ⍵ ←→ Value
     1=≢⍺:⍺ ⍺.Class.SetValue ⍵
     c←(0⊃⍺) GetComponent 1⊃⍺
     c c.Class.SetValue ⍵
 }

But we noticed GetComponent could be called with a reduction:

     c←GetComponent/⍺

and then why have the guard? As reduction on single item returns that item, we can just write:

SetComponentValue←{
     ⍝ ⍺ ←→ Component|Node and Name
     ⍝ ⍵ ←→ Value
     c←GetComponent/⍺
     c c.Class.SetValue ⍵
 }

Little things like this are nice.

Elias Mårtenson posts about a query in R and compares it to his APL-inspired language Kap. Elias of course provides a succint, APL-style solution in Kap. Here we take a look at the R solution and compare it to FlipDB, a relational database management system written in APL. The problem starts with the following table:

Given this, the task is to produce a table of total discounted sales by country, excluding outliers defined as entries in each country with an amount greater than 10 times the median for that particular country. The result we are looking for is:

We thus have a row-dependent where clause. That is, whether or not a row is included depends on other rows in the table. More precisely, whether or not a row is included in a group depends on the other rows in the group. This is the crux of the matter, and what makes the query problematic in some tools.

The R solution is given as:

  purchases |>
  group_by(country) |>   
  filter(amount <= median(amount) * 10) |>
  summarize(total = sum(amount - discount))

As the original post suggests, this is indeed a nice solution.

A comment on reddit makes the observation:

It's true that R's DSL has some nice defaults here, like the filtering happening implicitly on the grouped columns. But a DSL means there's stuff happening without straightforward execution semantics, there's some magic. And that example is very short in R, because it relies on those defaults, but if it needed sorting by total (instead of by country), or computing a flat amount-discount before grouping, then it'd start looking a bit longer.

We don't see the magic here. The R solution starts with a table or dataset in a column-store format, which is then partioned by country. In APL terms we can think of this as just making a vector of unique values from country and corresponding nested vectors out of amount and discount. All of the basic scalar operations are happy to work on nested vectors, applying scalar extension where necessary. We can then imagine sum and median being aggregate functions that have a built-in each operator when running on nested data. Same for filtering, where APL's replicate function would have a built-in each for nested data, taking a nested boolean array and masking the nested column data. DSLs can just be a collection of well-crafted, higher-level functions. If R were APL, the underlying code just jumps right out of this solution. It is simple, direct, executable and traceable step-by-step, unlike, say, an SQL query. And in fact, this is almost exactly how FlipDB works, and the solution is remarkably similar:

(Note that we have to explicitly specify the ordering which seems to be a default in R. We don't show the OrderBy clause in the queries below in the interest of space.)

The difference is that we are filtering in-line, just for one column in the result set, whereas the R solution is filtering the entire table. In-line filtering is useful because we can apply different filters, or no filter at all, for different columns in the result table. (Useful for a poor man's cross-tab.) For example, it might be useful to display the totals without excluding the outliers side-by-side for comparison which can be done by just adding another measure:

yielding:

Here we can see by inspection that only Canada has outliers.

Note that where is just a simple function that takes an array of one or more columns on the left and a corresponding boolean on the right and then returns the filtered columns. In this case, because we are just summing the result, we could replace where with multiplication, an age-old APL technique, zeroing out instead of filtering.

We can, like the R solution, specify a where clause for the query as a whole:

However, unlike the R solution, this is applied before, and independently of, grouping. Thus we need to specify a grouping in the where clause itself, and then apply the median function to each group using the by operator. The by operator handles the details of applying an aggregate function to grouped data, and then replicating the results to line up with the ungrouped data. The advantage here over the R solution is that we may then group our query by some other column or value than Country. For example, we might group by region, but keep outliers defined within country:

which yields:

What if we want to group by region but in addition display totals including the outliers, as we did above? We can't exclude that data from the query, but now we can't assume that the grouping for the query is the same grouping for computing the outliers. Rather than applying a where clause, we can pre-compute a column that flags outliers according to their country and then group based on region:

which yields:

In FlipDB computed columns in a query are executed before the where clause. This is useful, as we may want to reference them in the where clause, and if they are row-dependent we may want them computed on the entire table, not just what the where clause includes. However, perhaps in this case, it would be useful if computed columns were to execute after the where clause. We could have two sets of computed columns, one executed before the where clause and one executed after, but that seems a bit overkill.