Calibrate measurement rounding error SOLVED

[Mathew]

The "Calibrate Measurement" tool rounds off the measured distance too much (too few decimal places) so that the resulting preset is inaccurate.

For example, using snaps to make sure I got the distance exactly, I tried to set a measurement for the following:

image.png

But when I place the resulting dimension, it reads a different number, because of rounding error:

image(1).png

Would it be possible to increase the number of decimal places in "Distance Measured" in a future update?

ps. If I manually adjust the scale with more decimal places:

image(2).png

it shows correctly:

image(3).png

[Mathew]

btw, the precision is definitely there, but the tool is just rounding it off:

Code: Select all

{
let a=this.selectedAnnots[0];
let [p1,p2]=a.points;
console.println(Math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)/72);
}
// 0.6566666497124566

[Tracker Supp-Stefan]

Hello Mathew,

I will pass this on to our devs.
You can manually adjust the 0.66 to 0.6566 if needed (and as you have done later) however you are correct that a higher precision might indeed be beneficial there!
I've made the Feature request ticket in our internal system for that:
#6626: FR: Higher precision in the "Distance Measured" field in the Calibration tool
And I will be updating this topic with any new info I get in the ticket.

Kind regards,
Stefan

[Mathew]

You can manually adjust the 0.66 to 0.6566 if needed

I'm not able to get the preset to save the higher precision if I edit it, though: It just rounds it off to two decimal places. I can manually adjust an individual dimension line, and copy/paste (reuse) that one but it's not very efficient.

[Tracker Supp-Stefan]

Hello Mathew,

Thanks for the clarification! Indeed the preset also uses two decimal precision.
The "workaround" for now would be for you to e.g. multiply both sides of the preset by 100, and have 65.66 on one side and 1181 on the other side.

Kind regards,
Stefan

[Ivan - Tracker Software]

Please try the latest build (10.1.3.383) where we tried to fix the issue.

[Mathew]

I get the same result. IMO the "Distance Measured" should not be rounding at all. The "Name of the measure preset" makes sense to round.

image.png

image(1).png

image(2).png

I've attached a page from the pdf file I used.

pages from OPM-0394.pdf

I'd make a javascript tool to do this, but I don't see any way for javascript to access dimension scale.

[TrackerSupp-Daniel]

Hello, Mathew

Indeed it seems that the issue was only partially resolved, I am unsure as of yet whether the problem is specific to inches being used, or the process I am taking in changing the units in use, but there does still appear to be some rounding occurring which should not be. I do note that The accuracy is retained when calibrating with "points" instead of inches (likely due to points be a far more accurate unit), but what you need should be achieveable regardless of the unit selection. I have updated the ticket stefan created with two new gifs, showing the disparity between the two options.

PDFXEdit_Fi3eD0e9z6.gif

PDFXEdit_upIZgTX6tj.gif

As an aside, hopefully this knowledge helps you to circumvent the issue in the meantime.

Kind regards,

[Mathew]

Yes, using points increases the saved number to 4 significant figures. But then there's no option to override the automatically generated name of the measurement preset, so it becomes somewhat meaningless.

I think the problem is that the rounding is happening too soon (ie the programmer has rounded off the measurement before calculating the scale factor).

[TrackerSupp-Daniel]

Hello, Mathew

indeed, I have added a note in the ticket to increase the number of figured to 6 or 8 ideally, to allow for much higher accuracy of these measurements.

Kind regards,

[Mathew]

I just used this in build 385 and the precision has been increased, so it's now working correctly for me. Thank you.

[TrackerSupp-Daniel]