Scipy maintainer here, the main issue with the wheels was the Fortran77 that was SciPy throwing wrenches into the mix. With C/C++ self compilation should be quite straightforward. We (all Scientific Python packages) really worked hard on that.
From version 1.19 of SciPy there will be no need for fortran compilers (because we translated everything to C https://github.com/scipy/scipy/issues/18566) and then all becomes much easier in all platforms due to the large availability of C compilers in all platforms. Together with the Stable API developments in CPython the wheel clash issues "hopefully" will decrease gradually.
Thanks for all your hard work! I already consider SciPy and NumPy to be best practice in this area but as I'm sure you know, it's the long tail of stuff built on top of these and other core packages that are really the issue!!
Ah, my bad. what I failed to emphasize is that many of the downstream issues are coming from the upstream restrictions so this is one of the major blocks that was causing some mayhem down the line. So indirectly we might have caused some heartburn for you, apologies in advance.
> Actually the column-major order of Fortran is more efficient for some linear algebra operations than the order of C, which has been inherited by many modern languages that do not care about high performance in scientific computations.
This is a plausible assumption to make but unfortunately it is not true at large. Especially when the traditional sizes are exceeded say n >= 2000 certain operations such as LU can be improved in terms of performance with C-major arrays. However the correct statement is you lose at some place you win at other. There are certainly linalg operations that F-major can give you more performance. However it is also true for C-major layout.
In your example matrix vector product or any BLAS2 or BLAS3 level operations you can also swap out the for loop order to convert things around (row*col buffer multiplication vs sum of weighted column sum interpretation). In particular matrix norm operations are the only exceptions (abs column sum, row abs sum etc.) that certain norms prefer certain orders. In fact if you go into the Goto method deep enough you'll see that internal order is a bit like Morton ordering to fit things into L1 Cache.
The reason why column-major is preferred is historical and requires more surgery to get it running with C-major ordering. Trust me I tried but it's too much work to gain not so much. Maybe someday when I retire I can attempt it. Hence I kept it column major in my retranslation of LAPACK https://github.com/ilayn/semicolon-lapack
Instead I implemented a "high"-performance AVX2 matrix transpose operation so that swapping the memory layout is trivial compared to the linalg cost.
Read the sources carefully. Fortran quote is not his; he quoted it. Also remember that he was talking about pre-Fortran77 era. F77 tried to fix some of the criticisms though did not succeed fully. Here is a nice dig-in about the quote https://limited.systems/articles/dijkstra-fortran/
Another note to remember that John Backus, the team lead of the Fortran gang, was in the Algol committee. So these folks knew what they are talking about and spoke to each other periodically. Even John Backus said, Fortran is not the final interface that we should have.
It keeps spinning in the programming circles half-quoted versions of half-baked quotes from original sources. These pioneers, even when they disagreed, had pretty precise arguments and very rarely feeling the feelies.
FORTRAN —"the infantile disorder"—, by now nearly 20 years old, is hopelessly inadequate for whatever computer application you have in mind today: it is now too clumsy, too risky, and too expensive to use.
He told the truth and in turn Fortran corrected its course, but Dijkstra probably didn't change his mind about it.
---
> These pioneers, even when they disagreed, had pretty precise arguments and very rarely feeling the feelies.
The feud between Backus and Dijkstra kinda persuaded me of the opposite.
I'm working in the IIoT domain too. Your workflow is interesting towards the end. Any particular reason, why you don't write it to some db like Timescale or Influx at the end without any prometheus conversion?
Victoria Metrics is a Prometheus compatible time series database and was being used before I joined the company. I haven’t had any issues with it so I didn’t see a reason to pull it from our stack. Are you saying Timescale or Influx can natively ingest MQTT messages?
Similar to your mqtt converter you can make them ingest via Kafka or some other adapter in between. EMQX broker can directly write into TSDB but did not research into others whether they can or not.
Not the OTEL "telemetry" but sensors sending measurements from the field "telemetry". IT tech is, as usual, stealing perfectly defined engineering words and making them something else.
TimescaleDB is perfect if you also have relational data that you need to join with field data to the point that there no pros of using anything else for this use case, say you have 100000+ sensors and you need to group them by the customer site relations while aggregating per day statistics.
NumPy has no Fortran code, for quite a long time now. SciPy has and it is being rewritten. What you mention is the ufunc machinery underneath which is all C. NumPy also has SIMD support (albeit limited to certain functions). BLAS is also C/Assembly but only LAPACK is F77 (which is too much code to be rewritten).
This does not mean Fortran is bad (obligatory disclaimer for Fortran fans).
Also, Fortran has progressed immensely since F77 days. (I actually wrote some F77 code back in the day, when it was already ancient.) C has also progressed quite a bit since K&R days, but, to my mind, not nearly as much.
right, the problem for scipy developers I believe is that not enough of them know fortran, whereas C knowledge is necessary to hack on CPython native extensions.
I’m not going to lie, algebra wasn’t my cup of tea for a long time. Whenever you have a naive set of people commenting on a thread, my theory is that there is a category of folks who always take it too far leading to people feeling left out. An excluded middle if you will. I say education is at the root of the problem, and blame the state machine for that.
Same experience for me. I was able to buy my first drumset from the money I got for making a PHP+MySQL+HTML website for someone (also done all in notepad). I did not know anything about computers but I needed to buy a drumset. And that page actually got me going about how HTML works.
I still remember their animations about car differential which were magical.
From version 1.19 of SciPy there will be no need for fortran compilers (because we translated everything to C https://github.com/scipy/scipy/issues/18566) and then all becomes much easier in all platforms due to the large availability of C compilers in all platforms. Together with the Stable API developments in CPython the wheel clash issues "hopefully" will decrease gradually.